30-2214-01  (00  JAN  70) 


METRICS  OF  SOFTWARE  QUALITY 
FINAL  TECHNICAL  REPORT 


MCOOWWELL 

DOUGLAS  (3 

com 


>/v 


NOVEMBER  1980 


MDC  G9326 


AIR  FORCE  OFFICE  OF  SCIENTIFIC  RESEARCH  (AFSC) 
NOTICE  OF  TRANSMITTAL  TO  DDC 
This  technical  report  has  been  reviewed  and  is 
approved  for  public  release  IAW  AFR  190-12  (7b) 
Distribution  is  unlimited. 

A.  D.  BLOSB 

rechnioal  Infonaation  Officer 


MCDONNELL  DOUGLAS  ASTRONAUTICS  COM  RANT- HUNTINGTON  BEACH 

5301  Bolsa  Avenue  Huntington  Beach,  California  92647  1714)  896-3311 


McDonnell  Douglas  Astronautics  Company 
5301  Bolsa  Avenue 
Huntington  Beach,  CA  92647 


tl.  CONTROLLING  OFFICE  NAME  AND  ADDRESS  /  /» 

Air  Force  Office  of  Scientific  Research  / // 
Bolling  Air  Force  Base  ( 

District  of  Columbia  20332 


4.  MONITORING  AGENCY  NAME  A  ADDRESS  (tf  dl/farant  from  Controlling  Office)  15.  SECURITY  CLASS.  ( o , 


aliWBiESSCTaBBMiM 


Unclassified 


15«.  OECLASSIFICATION/ 
SCHEDULE 


16.  DISTRIBUTION  STATEMENT  (of  thfa  Raport) 


Approved  for  public  release;  distribution  unlimited 


17.  DISTRIBUTION  STATEMENT  (of  tha  abatrmct  ontorod  In  Block  30,  If  dUfatant  from  Roport) 


If.  KEY  WORDS  (Contlnua  on  rmvaraa  a/da  If  nacaaaary  and  Idanttfy  by  block  number) 

Software  Metrics 
Test  Tools 


20.  ABSTRACT  fCortllnu*  on  ravoraa  alda  If  nacaaaary  and  fdontlty  by  block  numbar) 

x^This  report  covers  the  period  from  1  June  1977  to  30  October  1980. 

A  major  task  on  this  contract  was  to  make  a  comprehensive  review  of 
the  literature  on  software  metrics  and  of  quantitative  measures  of  ' 

(Continued  on  next  page) 


DD  1473 


COITION  or  I  NOV  »S  1$  obsolete 


3 


security  classification  of  this  page  rm*n  Dtt*  Cniwmi 


SICUH1TY  CLASSIFICATION  OF  THIS  FAOtfhl  Ml  tnfnd) 


program  testing.)  The  original  review  is  contained  in  the  first  Interim 
Report  (MDC  G75T7,  dated  July  1978);  this  review  has  been  slightly  revised 
and^updateci  in  this  report. 

-^Another  accomplishment  was  the  development  of  an  automatic  procedure  for 
testing  FORTRAN  programs  with  random  numbers  and  random  symbols.  This 
procedure  first  drives  the  program  with  sets  of  random  data,  then  senses 
the  tracks  taken,  compares  each  generated  track  against  all  predecessors, 
then  on  the  basis  of  the  pattern  of  occurrence  of  new  tracks,  provides  an 
estimate  of  the  total  number  of  residual  paths.  Additional  sets  of  random 
numbers  can  be  then  generated. 

The  program  developed  to  instrument  a  given  FORTRAN  program  and  provide 
data  for  evaluating  coverage  is  described  in  the  Final  Report  of  earlier 
work  (MDC  G6553,  dated  December  1976).  Changes  which  have  been  made  are 
described  herein. 

In  the  related  topics  of  Software  Reliability,  two  methods  of  estimating 
the  residual  error  content  of  an  entire  program  on  the  basis  of  data 
obtained  in  the  testing  of  portions  of  it  have  been  developed  and  are 
detailed  here. 
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This  report  documents  the  results  obtained  during  an  Air  Force 
Office  of  Scientific  Research  Contract  entitled 

"Metrics  of  Software  Quality."  This  work  conducted  during  the 
period  1  June  1977  to  31  October  1980,  was  performed  by  personnel 
of  the  Computer  Science  Branch  of  the  Data  Control  and  Processing 
Subsystems  Department  of  Av'onics  Control  and  Information  Systems 
Subdivision,  McDonnell  Douglas  Astronautics  Company-West,  in 
Huntington  Beach,  California.  The  Principal  Investigator  and  study 
director  was  Zygmund  Jel inski.  Substantial  contributors  to  the  study 
were  P.  B.  Moranda  and  J.  B.  Churchwell .  This  work  was  monitored  by 
Lt.  Col.  George  W.  McKemie  whose  assistance  is  gratefully  acknowledged. 
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A  major  task  on  this  contract  was  to  make  a  comprehensive  review 
of  the  literature  on  software  metrics  and  of  quantitative  measures  of 
program  testing.  The  original  review  is  contained  in  the  first 
Interim  Report  (MDC  G7517,  dated  July  1978);  this  review  has  been 
slightly  revised~~a rid' updated  in  this  report.  . 
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Another  accomplishment  was  the  development  of  an  automatic  pro¬ 
cedure  for  testing  FORTRAN  programs  with  random  numbers  and  random 
symbols.  This  procedure  first  drives  the  program  with  sets  of  random 
data,  then  senses  the  tracks  taken,  compares  each  generated  track 
against  all  predecessors,  then  on  the  basis  of  the  pattern  of  occur¬ 
rence  of  new  tracks,  provides  an  estimate  of  the  total  number  of 
residual  paths.  Additional  sets  of  random  numbers  can  be  then 
generated. 

The  program  developed  to  instrument  a  given  FORTRAN  program  and 
provide  data  for  evaluating  coverage  is  described  in  the  Final  Report 
of  earlier  work  (MDC  G6533,  dated  December  1976).  Changes  which  have 
been  made  are  described  herein. 

In  the  related  topics  of  Software  Reliability,  two  methods  of 
estimating  the  residual  error  content  of  an  entire  program  on  the 
basis  of  data  obtained  in  the  testing  of  portions  of  it  have  been 
developed  and  are  detailed  here. 
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Section  1 

INTRODUCTION  AND  OVERVIEW 

1.1  INTRODUCTION 

The  essential  focus  in  this  research  has  been  the  production  of  useful 
metrics  to  characterize  the  quality  of  software  at  any  stage  of  its  develop¬ 
ment.  The  end  product  desired  is  simply  a  set  of  metrics  which  have  utility 
and  a  description  of  the  means  of  applying  them.  The  metrics  selected  are 
few  in  number.  For  completeness  a  list  of  all  metrics  which  were  found  and 
considered  as  candidates  at  the  time  of  the  review  in  1978  are  included  with 
comments  on  their  prospective  use. 

The  primary  metric  considered  as  an  integral  part  of  this  research  is 
coverage.  Coverage  can  be  described  by  a  spectrum  of  choices:  coverage  at 
the  branch  level,  at  the  instruction  level,  the  segment  level,  "track"  level, 
or  execution  path  level.  The  procedures  which  are  required  to  firmly  estab¬ 
lish  the  level  of  testedness  or  coverage  of  a  software  package  form  the 
substantive  portion  of  this  research. 

In  addition  to  coverage  metrics,  the  software  reliability  models 
developed  during  an  earlier  contract  and  which  have  been  employed  to  estimate 
mean  times  to  failure  and  error  content  of  a  completed  package,  were  modified 
so  as  to  derive  estimates  on  the  basis  of  observations  of  errors  during  the 
initial  phases  of  testing  on  incomplete  or  partial  programs.  Two  models  were 
developed,  one  is  based  on  the  assumption  that  a  cumulative  record  is  main¬ 
tained  of  the  percentage  of  the  completed  program  that  the  (varying)  tested 
portion  represents,  the  other  assumes  that  the  count  of  the  number  of 
instructions  under  test  is  available. 


1.2  OBJECTIVES  AND  TASKS  DESCRIPTIONS 

The  original  plan  for  the  first  phase  of  the  research  was  given  in  terms 
of  the  tasks: 

A.  Review  contemporary  work  of  researchers  in  software  testing  field  to 
postulate  testing  strategies. 
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8.  Perform  preliminary  tests  of  selected  programs  to  obtain  some  data 
on  various  testing  strategies. 

C.  Evaluate  parameters  influencing  software  quality  to  suggest 
appropriate  metrics. 

D.  Document  as  appropriate  to  facilitate  later  extensive  experiments. 

The  primary  effort  during  the  second  period  encompassed  work  on  three 
tasks: 

A.  Tailor  or  expand  the  testing  programs  that  were  developed  in  the 
first  phase  of  the  contract. 

B.  Code  so  as  to  provide  valuations  of  the  program  predicates,  and 
values  of  the  artificial  program  variables  which  provide  the  data  for  the 
search  procedures. 

C.  Modify,  install,  and  test  the  tool  on  a  laboratory  computer  when 
the  scope  and  size  of  the  test  tool  are  established. 

The  subsequent  work  centered  on  the  assessment  of  the  practicality  of 
a  fully  automated  version  of  testing.  The  goal  of  this  work  was  to  test  the 
tool  and  the  methodology,  using  the  several  constructs  (connection  matrix, 
status  vectors,  predicate  valuations,  and  input  and  output  data)  through  the 
implementation  on  a  "laboratory"  type  of  computer,  such  as  the  Nanodata  QM-1 . 
The  assessment  of  the  practicality  has  been  carried  out  but  the  implementa¬ 
tion  required  programming  effort  which  far  exceeded  budgeted  labor. 

1.3  COURSE  OF  THE  RESEARCH  PROGRAM 

The  initial  effort  was  made  on  a  literature  review.  The  review  made  was 
very  comprehensive  and  its  scope  is  indicated  by  the  list  of  papers,  publi¬ 
cations,  and  reports  which  were  reviewed  either  in  depth  and  critically,  or 
in  content.  This  list  comprises  Section  2.1.  Subsequently,  evaluations  of 
the  metrics  which  had  potential  ability  were  made.  These  are  given  in 
Section  2.2. 
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The  substantive  contribution  to  two  aspects  of  software  quality  were 
initiated  after  the  literature  review.  The  metrics  which  showed  promise, 
the  two  which  could  be  most  productively  studied  were  the  degree  of  coverage 
(or  testedness)  and  MTTF  (meantime  to  next  error  detection).  These  two 
metrics  were  exclusively  studied  in  this  research. 

Previous  successes  in  testing  simple  programs  with  random  numbers  led 

to  the  belief  that  such  testing  could  serve  as  an  efficient  initial  testing 

set  on  more  complex  programs.  This  testing  has  been  found  to  be  very  much 

more  effective  than  the  sometimes  trivial  and  always,  limited  in  number, 

check  problems,  which  are  normally  used  (some  of  the  cases  generated  by 

random  test  data  for  a  polynomial -sol ving  routine  produced  polynomials  whose 

1 3 

coefficients  had  ratios  of  10  and  this  would  be  an  essentially  incon¬ 
ceivable  choice  for  most  testers).  Accordingly,  experiments  were  performed 
on  progressively  larger  FORTRAN  programs  and  the  general  tendency  which  was 
indicated  on  smaller  programs  was  supported,  random  numbers  generally  pro¬ 
duced  good  tests  and  in  fact,  in  one  of  the  later  programs  tested,  the  first 
100  random  number  sets  produced  99  different  tracks.* 

A  means  of  determining  the  number  of  residual  tracks  still  existing  has 
been  developed  in  earlier  work  (Reference  1)  but  it  required  a  time  consuming 
matching  or  comparison  procedure.  One  of  the  earliest  programming  tasks 
was  to  automate  this  procedure.  Developed  was  an  even  further  extension  of 
the  already  augmented  Program  Evaluation  and  Tester  (PET)  tool,  which  itself 
was  developed  in  1974  (Reference  2).  With  the  random  number  generator,  which 
was  an  augmentation  to  PET  made  in  earlier  work  (Reference  1),  and  distin¬ 
guished  by  the  name  Program  Testing  System  (PTS),  and  the  automated  compari¬ 
son  procedure,  it  was  possible  to  produce  sets  of  data  in  quantities  approach¬ 
ing,  to  any  desired  degree,  the  asymptotic  number  of  tracks  which  could  ever 
be  generated  by  random  numbers.  The  modified  program  is  designated  as  the 
Augmented  Program  Testing  System  (APTS). 


*A  track  derives  its  name  from  two  facts:  first,  a  segment  of  the  associated 
execution  sequence  is  counted  only  once  even  though  it  may  have  appeared  with 
multiplicity  in  the  execution  sequence;  second,  the  order  of  execution  is 
immaterial  and,  for  example,  the  sequences  ABCAC  and  BACCA  are  equivalent. 
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The  next  step  was  to  develop  a  systematic  way  of  executing  the  still 
remaining  tracks  by  formation  of  what  are  called  constructed  cases.  The 
framework  in  this  was  a  procedure  developed  by  W.  Miller  and  D.  L.  Spooner 
(Reference  3).  This  consists  in  converting  the  problem  of  unguided  search¬ 
ing  for  data  input  points  which  will  drive  a  particular  path  or  track,  into 
a  systematic  process  related  to  a  common  optimization  problem.  Auxiliary 
variables  are  inserted  at  predicate  sites,  and  a  certain  simple  function  of 
these  variables  is  evaluated  for  each  input  data  set,  when  a  data  set  is 
found  which  produces  a  positive  value  in  the  function,  the  path  associated 
with  the  preassigned  predicate  valuations  then  will  have  been  exercised; 
and,  if  the  trial  data  set  does  not  produce  a  positive  value,  then  any 
of  several  search  techniques  commonly  employed  in  optimization  problems  can 
be  used  to  determine  subsequent  trial  data  sets.  The  initial  work  in  this 
application  was  all  performed  without  computer  assistance. 

The  final  major  goal  was  to  incorporate  both  the  random  testing  and 
constructed  cases  into  one  comprehensive  sequential  testing  process.  Starting 
with  random  numbers,  these  would  be  employed  until  new  cases  become  difficult 
to  find,  at  which  point  a  transfer  to  testing  by  constructed  cases  would  be 
made,  using  displays  to  show  the  predicate  sites  and  the  unexercised  branches 
(valuations  of  the  predicates).  Auxiliary  variables  would  then  be  inserted, 
the  program  would  be  recompiled,  starting-  or  trial-data  would  be  used,  the 
composite  objective  function  would  be  evaluated,  and  a  search  procedure 
invoked.  The  latter  two  steps  would  be  carried  out  in  an  iterative  fashion 
until  the  selected  path  was  achieved  or  judged  to  be  infeasible.  Unfortu¬ 
nately,  the  magnitude  of  the  programming  effort  required  to  implement  the 
display/operator/computer  complex  was  judged  to  be  too  extensive  and  expensive 
to  carry  out.  Accordingly,  only  portions  of  the  implementation  have  been 
developed.  These  are  described  in  this  report. 

With  respect  to  the  study  of  the  MTTF,  it  was  carried  out  in  a  low 
priority  status,  throughout  the  course  of  the  research.  Interfaces  were  made 
at  several  conferences  with  most  of  the  analysts  who  have  worked  in  the  field 
of  software  error  modelling.  Two  significant  new  models  of  the  error-making 
process  were  developed  during  the  third  year  of  this  study. 
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1.4  PUBLICATIONS  AND  PRESENTATIONS 

The  following  are  the  major  publications  or  presentations  of  research 

sponsored  in  whole  or  in  part  by  the  Air  Force  Office  of  Scientific  Research 

1.  P.  B.  Moranda,  "Limits  to  Program  Testing  with  Random  Number  Inputs", 
Proceedings  of  COMSAC  1978,  November  13-15,  Chicago,  Illinois. 

2.  P.  B.  Moranda,  "Event-Altered  Rate  Models  for  General  Reliability 
Analysis",  IEEE  Transactions  on  Reliability,  Vol  R-28,  No.  5, 

December  1979. 

3.  Presentation  by  P.  B.  Moranda  of  a  paper,  "On  the  Modelling  of  the  Error 
Process",  to  the  1st  Minnowbrook  Workshop  on  Software  Performance 
Evaluation,  sponsored  by  Syracuse  University  at  Rome  Air  Development 
Center,  October  1978. 

4.  P.  B.  Moranda,  "Error  Detection  Models  for  Application  During  Program 
Development",  Proceedings  of  Pathways  of  System  Integrity,  ACM  Meeting 
Gaithersbu ry,  MD,  June  1980. 

5.  Presentation  by  P.  B.  Moranda  of  same  paper  at  National  Computer 
Conference,  Anaheim,  California,  22  May  1980. 

6.  Presentation  by  P.  B.  Moranda  of  same  paper  to  3rd  Minnowbrook  Workshop 
on  Software  Performance  Evaluation,  sponsored  by  Syracuse  University  and 
Rome  Air  Development  Center,  August  1980. 

7.  Presentation  by  Z.  Jel inski  of  a  paper  "An  Approach  to  Solution  of 
Problems  with  Support  Software  as  Deliverables"  to  Defense  Systems 
Management  Review,  Ft.  Belvoir,  Virginia,  March  1978. 

8.  P.  B.  Moranda,  "Asymptotic  Limits  to  Program  Testing,  INFOTECH 
State-of-the-Art  Report  on  Program  Testing,  INFOTECH  1979. 
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Section  2 

LITERATURE  REVIEW  AND  CRITIQUES  OF  SOFTWARE  METRICS 


2.1  REVIEWS 

Two  different  levels  of  review  were  made,  one  is  thorough  and  complete 
at  an  analytical  level,  the  other  for  content  only. 


2.1.1  In  Depth  Reviews 


2. 1.1.1  Review  of  Work  Published  Prior  to  July  1978 

Rather  extensive  and  detailed  examinations  were  made  of  the  literature 
of  the  software  testing  field  and  of  software  metrics  in  general.  The 
following  papers  were  reviewed  indepth  during  the  period  June  1977  to 
June  1978.  Informal  reviews  of  the  following  listed  papers  were  provided 
to  the  contractor. 

1.  TRW  Software  Reliability  Study.  TRW  Final  Report,  RADC,  TR-76-236, 

August  1976. 

2.  M.  L.  Shooman,  "Structural  Models  for  Software  Reliability  Predictions", 
Proceedings  of  the  2nd  International  Conference  on  Software  Engineering, 
13-16  Oct  1976,  San  Francisco,  California. 

3.  H.  E.  Williams,  T.  A.  James,  A.  A.  Beaureguard,  and  P.  Hilcoff, 

"Software  Reliability  Systems:  A  Raytheon  Project  History", 
RADC-TR-77-188,  Final  Technical  Report,  June  1977. 

4.  IBM  Federal  Systems  Division,  "Statistical  Prediction  of  Programming 
Errors",  RADC-TR-77-175  Final  Technical  Report,  May  1977. 

5.  Doty  Associates,  Inc.,  "Software  Cost  Estimation:  Vol  1",  RADC-TR-77-220, 
Final  Technical  Report,  June  1977. 

6.  J.  R.  Brown,  H.  N.  Buchanan,  "The  Quantitative  Measurement  of  Software 
Safety  and  Reliability"  SDP  1776,  24  August  1973. 

7.  M.  Shooman  and  A.  Laemmel  "Statistical  Theory  of  Computer  Programs  - 
Information  Content  and  Complexity"  Digest  of  Papers  Fall  COMPCON  77, 
Washington,  0.  C.,  6-9  September  1977. 

8.  G.  J.  Schick  and  R.  W.  Wolverton,  "An  Analysis  of  Competing  Software 
Reliability  Models"  IEEETSE,  March  1978;  Vol.  SE-4,  No.  2. 
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9.  G.  J.  Myers,  Software  Reliability,  Wiley-Interscience,  1976. 

10.  A.  Fitzsimmons  and  T.  Love,  "A  Review  and  Evaluation  of  Software 
Science",  ACM  Computing  Surveys  Vol .  10,  No.  1,  March  1978. 

11.  B.  Littlewood  and  J.  L.  Verrall,  "A  Bayesian  Reliability  Growth  Model 
for  Computer  Software",  Record  of  1973  Symposium  on  Computer  Software 
Reliability,  New  York,  N.Y.  ,  1973. 

2. 1.1. 2  Additional  Reviews 

1.  A.  L.  Goel  and  K.  Okumoto,  "Bayesian  Software  Prediction  Models,  Vol  I : 

An  Imperfect  Debugging  Model  for  Reliability  and  Other  Quantitative 
Measures  of  Software  Systems",  RADC-TR-78-1 55,  Rome  Air  Development 
Center,  N.Y. ,  1978. 

2.  J.  D.  Musa,  "Progress  in  Software  Reliability  Measurements",  Proc.  2nd 
Software  Life  Cycle  Management  Workshop,  Atlanta,  Georgia,  August  1978. 

3.  R.  E.  Schafer,  et  al,  "Validation  of  Software  Reliability  Models",  Hughes 
Aircraft  Co.,  RADC-TR-79-147,  June  1979. 

4.  W.  D.  Brooks,  R.  W.  Motley,  "Analysis  of  Discrete  Software  Reliability 
Models",  IBM  Corp. ,  RADC-TR-80-84,  RADC,  New  York,  April  1980. 

5.  E.  H.  Forman  and  N.  0.  Singprrwalla,  "An  Empirical  Stopping  Rule  for 
Debugging  and  Testing  Computer  Software",  Journal  of  the  American 
Statistical  Association,  Vol  72,  December  1977. 

6.  A.  N.  Sukert,  "An  Investigation  of  Software  Reliability  Models",  Proc. 
1977  Annual  Rel .  Maint.  Symp.,  Philadelphia,  PA,  1977. 


2.1.2  Literature  Reviewed  for  Content 

1.  Z.  Manna.  Mathematical  Theory  of  Computation,  McGraw-Hill,  Inc., 

New  York,  1974. 

2.  T.  Gilb,  Software  Metrics,  Winthrop  Publishers,  Inc.,  Cambridge, 

Mass.,  1977. 

3.  A.  Goel.  "Bayesian  Software  Predictions  Models,"  RADC-TR-77-112, 

March  1977. 

4.  M.  Shooman,  "Manpower  Deployment  Effects  on  Software  Error  Models," 
in  RADC-TR-76-143,  May  1976. 

5.  Boeing  Computer  Services,  "Software  Data  Acquisition,"  RADC-TR-77-1 30, 
April  1977. 

6.  W.  H.  Howden,  "Methodology  for  Generation  .of  Program  Test  Data," 

IEEE  TransComp,  Vol.  C-24,  May  1975. 

7.  L.  Clarke,  "A  System  to  Generate  Test  Data  and  Symbolically  Execute 
Programs,"  IEEETSE ,  SE-2,  1976. 
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8.  S.  Gerhart  and  L.  Yelowitz,  "Fallibility  in  Applications  of  Modern 
Programming  Techniques,"  IEEETSE  Vol .  SE-2,  No.  3,  Sept.  1976. 

9.  R.  F.  Serfozo,  "Compositions,  Inverses,  and  Thinning  of  Random 
Measures,"  Syracuse  University,  Dept,  of  Ind.  Eng.  and  Ops.  Research, 
December  1975. 

10.  L.  Osterweil,  "Depth-First  Search  Techniques  and  Efficient  Methods  for 
Creating  Test  Paths,"  Univ.  of  Colorado  Dept,  of  Comp  Sci  TR  No. 
CU-CS-077-75,  August  1975. 

11.  W.  Miller  and  D.  Spooner,  "Automatic  Generation  of  Floating  Point  Test 
Data,"  IEEETSE  Vol.  SE-2,  No.  3,  Sept.  1976. 

12.  G.E.P.  Box  and  K.  B.  Wilson,  "Attainment  of  Optimum  Conditions," 

J.  Royal  Stat  Soc.,  Vol.  XIII,  No.  1,  1951. 

2.1.3  Review  of  Testing  Tools  and  Procedures 

Articles  of  a  review  nature  have  identified  and  briefly  described  a 
large  number  of  different  testing  tools.  D.  J.  Reifer  (Reference  4) 
identified  70  different  types  of  tools  and  briefly  discussed  each  type. 

C.  V.  Ramamoorthy  and  S.  F.  Ho  (Reference  5)  discuss,  in  some  detail,  15 
different  tool  types.  A  review  of  these  different  types  here  would  be  dupli¬ 
cative.  Instead  a  composite  review  of  the  limited  number  of  reports  listed 
above  dealing  with  the  testing  process  will  be  presented.  Usually  the 
potential  deficiencies  of  the  processes  or  tools  are  brought  out  in  the 
description,  but  not  their  advantages. 


Before  the  discussion  of  individual  classes  of  tools  is  undertaken  here, 
it  is  well  to  note  that  the  paper  by  Goodenough  and  Gerhart  (Reference  6) 
illuminates  many  of  the  heretofore  neglected  points  concerning  testing. 


Some  of  the  important  points  they  make  in  this  respect  are: 

A.  It  is  not  enough  to  execute  a  statement  with  a  particular  set  of 
conditions,  it  must  be  tested  in  all  combinations  of  conditions; 

B.  In  the  same  sense,  a  path  through  a  loop  may  have  to  be  taken 
several  times  before  the  conditions  for  error  revelation  are  met; 

C.  Missing,  but  requi red-for-correctness ,  components  of  a  program 
(such  as  predicates  or  assignments)  clearly  cannot  be  identified  by 
"cover-testing"  a  program; 

D.  Generally  a  program  must  be  examined  for  what  it  actually  does 
instead  of  what  the  tester  is  told  the  program  does  and,  at  each  point  of 
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interface,  it  must  be  examined  for  what  it  can  do; 

E.  The  environment,  including  the  operating  system,  hardware 
processor  and  language,  have  to  be  examined. 

2. 1.3.1  Inside  Out  Testing 

Several  different  techniques  have  been  employed  to  develop  test  cases  on 
the  basis  of  a  specified  set  of  valuations  or  outcomes  of  the  program's 
predicate.  The  mathematical  expressions  employed  in  the  program  predicates, 
are  used  to  develop  a  set  of  restrictions  on  the  input  data  space.  Solution 
of  the  set  of  equations  then  produces  a  point  or  set  of  points  that  will 
achieve  the  path  through  the  program.  The  difficulty  with  the  procedure  is 
that  the  set  of  equations  involved  often  are  not  tractable,  even  for  cases 
where  only  "area"  (as  distinct  from  point)  solutions  are  required.  This 
difficulty  is,  to  a  degree,  alleviated  by  use  of  interactive  entry  of  data 
and  display-guided  solutions. 

2. 1.3. 2  Symbolic  Execution 

Instead  of  operating  on  numerical  (or  logical)  values  for  variables  in 
a  processing,  the  program's  operations  can  be  carried  out  on  the  symbols 
themselves.  This  technique  was  independently  proposed  by  W.  E.  Howden 
(Reference  7)  at  McDonnell  Douglas  Astronautics  Company,  B.  Elspas,  et  al . 
(Reference  8)  at  Stanford  Research  Institute,  J.  C.  King  of  IBM  (Reference  9) 
and  Lori  A.  Clark  (Reference  10)  of  the  University  of  Massachusetts. 

Programs,  so  exercised  must  be  augmented  so  they  become  capable  of 
symbolic  execution  of  expressions  and  provide  means  for  selecting  specified 
branches  or  paths  in  them.  Howden  employs  a  system  (DISSECT)  processing  the 
program  that  is  to  be  symbolically  executed,  along  with  a  list  of  commands 
that  cause  symbolic  execution. 

The  advantages  of  symbolic  execution  are  clear.  In  certain  cases  the 
printout  consists  of  an  explicit  formula  that  is  unambiguous  to  the  reader. 

If  the  formula  is  correct,  the  program  is  correct  for  all  data  and  there  is 
no  necessity  for  numerical  comparisons  or  independent  checks. 

In  many  cases,  however,  the  output  is  far  from  clear  to  any  but  the  most 
experienced  users.  There  is,  for  example,  sometimes  a  need  to  maintain  the 
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list  of  possible  antecedents  (a  suspense  file)  for  a  program  variable  having 
several  different  symbols  and  values  assigned  to  it.  Further  there  is  a 
context-dependency  that  a  given  assignment  may  have,  caused,  for  example, 
by  different  encounters  of  an  assignment  during  looping.  This  must  be 
accounted  for,  and  in  the  case  of  DISSECT,  the  context  is  identified  by  a 
number  representing  the  dynamic  instruction  number  (as  distinct  from  the 
static  sequence  number  associated  with  a  listing).  These  and  more  complex 
problems  have  been  faced  by  Howden  and  others  and  they  provide  finished 
products  that  are  proof  that  sucH  techniques  can  be  used  to  good  effect 
when  the  tools  are  in  the  hands  of  experts. 

While  there  are  some  barriers  to  the  "field"  use  of  such  techniques, 
they  do  not  seen  insurmountable  and  it  is  probably  reasonable  to  expect  that 
symbolic  execution  can  be  of  common  use. 

2. 1.3. 3  Automated  Verification  Systems 

Several  systems  instrumenting  a  given  program  to  permit  the  tallying  of 
the  uses  of  its  instructions,  branches,  and  so  forth,  are  classified  as 
automated  verification  systems  by  Reifer  (Reference  4).  They  are  usually  not 
automated  in  the  strict  sense  of  the  word:  although  they  require  a  set  of 
input  test  data  to  drive  the  program,  there  is  no  instantaneous  feedback  to 
change  the  data  to  test  new  unexercised  sections  of  the  program.  A  complaint 
on  word  usage  can  be  also  made  that  these  systems  do  not  really  verify  the 
tested  program,  and  generally  do  not  even  consider  the  output  in  respect  to 
its  accuracy,  or  even  its  relevance. 

A  McDonnell  Douglas  Automation  Company  tool,  called  PET  (for  Program 
Evaluator  and  Tester)  described  by  L.  6.  Stucki  in  a  company  report 
(Reference  2)  and  in  the  open  literature  (Reference  11),  is  typical  of  this 
class. 

For  a  given  data  set,  PET  reports  the  usage  by  instruction  and  branch, 
which  the  execution  sequence  represents.  There  are  other  useful  metrics, 
including  the  range  of  value  for  each  of  the  program  variables.  Lists  of 
unexercised  program  components  also  are  printed  out. 

An  augmented  version  of  PET,  that  formed  segments  consisting  of 
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‘  "dynamically  contiguous"  program  instructions,  was  used  and  described  in  a 

recent  AFOSR-sponsored  study  (Reference  1).  In  that  study,  as  with  most 
other  applications  of  PET,  the  emphasis  is  on  the  "coverage"  of  the  tested 
program.  Repeated  tests  with  randomly  generated  input  data  were  used,  and 
their  effects  merged  to  produce  a  composite  (montage)  of  the  testing  status 
of  the  program.  Unexercised  segments  were  used  to  find  the  governing  predi¬ 
cate  or  predicates  in  the  program  listing,  and  so-called  constructed  cases 
were  then  formed.  The  process  was  continued  as  far  as  deemed  possible  to 
establish  the  testing  degree. 

This  class  of  program  monitors  is  useful  in  another  way.  Frequently 
exercised  portions  of  a  program  can  be  identified  by  the  tallies  or  counts 
and  the  identified  regions  can  be  examined  to  see  if  improvements  can  be 
made  in  the  coding  or  basic  algorithms. 

2. 1.3. 4  Automatic  Test  Generators 

Conceivably  any  particular  segment  of  a  program  has  some  input  data  that 
will  cause  it  to  be  exercised.  Sines  it  is  possible,  as  indicated  in  the 
section  on  inside-out  testing,  to  back  up  from  a  particular  point  in  the  pro¬ 
gram  to  the  "top,"  it  should  be  possible  to  choose  a  set  of  inputs  that  will 
cause  any  given  segment  to  be  exercised.  The  technique  used  amounts  to  an 
identification  of  the  program  variables  that  are  "active"  at  the  segment,  and 
then  to  relate  these  to  the  input  variables.  This  is  illustrated  in 
Section  3.1.2  where  the  precise  set  of  relations  to  the  input  data  are 
developed  explicitly  from  a  particular  "straight  line"  path  through  the 
program. 

Usually  it  will  not  be  necessary  to  develop  the  precise  relations  (which, 
it  is  noted,  is  almost  the  same  as  symbolic  execution)  between  the  program 
variables  and  the  input,  and  it  is  only  necessary  to  identify  those  inputs 
affecting  the  selected  program  variable.  This  can  be  accomplished  in  an  even 
less  elegant  way  by  simply  generating  random  numbers  to  serve  as  values  for 
the  input  variables. 

Whatever  scheme  is  used,  the  automatic  test  generators  provide  a  basis 
for  economical ly  meaningful  testing-to-"completion. "  The  idea  is  simply  to 
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form  a  "feedback"  loop  between  a  cumulative  record  of  the  segments  previously 
tested  to,  what  might  be  called,  a  scenario  generator.  The  scenario 
generator  would  provide  a  one-at-a-time  selection  for  the  untested  segments, 
and  the  standard  test  generators  could  be  used  to  "find"  the  required  data. 
This  will  then  cause  the  new  scenario  update  and  a  new  selection.  This  idea 
is  mentioned  again  later  in  connection  with  the  use  of  "tracks"  and  the 
automatic  case  generation  process. 

2. 1.3. 5  Domain-Testing  Strategies 

The  point  mentioned  above,  in  connection  with  the  possible  creation  of  a 
truly  automatic  test  tool,  brings  up  the  important  problem  identified  earlier, 
the  essential  impossibility  of  producing  a  particular  numerical  value  by  the 
usual  kinds  of  random  number  generation.  This  is  not  a  problem  in  estimating 
the  asymptotic  limits  to  testing  with  such  numbers,  because  of  the  infrequent 
occurrence  of  these  numbers  in  the  sample.  For  tne  development  of  con¬ 
structed  cases  where  exhaustive  testing  can  be  achieved,  it  is  necessary  to 
specify  the  set  of  points  in  the  input  space  which,  after  processing,  will 
produce  a  specified  value  for  a  program  variable. 

Generally  speaking,  the  particular  set  of  points  achieving  the  specified 
value  has  relatively  small  dimensionality  (a  point  in  two-space,  a  line  in 
three-space,  etc.)  making  the  problem  of  testing  boundaries  important. 

E.  I.  Cohen  and  L.  J.  White  of  Ohio  State  University  (Reference  12),  have 
investigated  this  and  similar  problems  and  developed  strategies  that  will  test 
domains  with  linear  and  non-linear  boundaries  (the  latter  only  in  two 
dimensions  at  present)  in  efficient  ways.  As  noted,  work  of  this  kind  is 
essential  to  any  ultimately  automatic  testing  scheme. 

2.2  CRITIQUES  OF  SOFTWARE  METRICS 

2.2.1  Software  Science  Metri cs 

In  the  review  paper  by  A.  Fitzsimmons  and  T.  Love  (Reference  13)  the 
principal  metrics  employed  in  Software  Science  are  discussed  in  some  detail. 
They  are  few  in  number:  length,  volume,  program  level,  language  level, 
effort  and  time. 
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A  troubling  feature  of  these  metrics  is  that  they  are  all  based  on  counts 
of  operators  and  operands  and,  as  noted  in  Reference  14,  there  are  many  cases 
where  it  is  not  at  all  clear  what  particular  mix  of  these  fundamental  elements 
a  given  program  instruction  represents.  The  effects  of  this  lack  of  precision 
in  the  definition  of  operator  and  operand  has  been  studied  by  J.  L.  Elshoff 
(Reference  15).  In  this  study  all  of  the  primary  metrics  are  computed  for 
some  34  different  programs  for  each  of  eight  different  interpretations  of  the 
way  in  which  the  counts  of  programming  elements  should  be  taken.  These  eight 
different  methods  produced  exceptional  variability  in  the  metrics  in  cases 
where  there  was  a  significant  effect  in  the  vocabulary  definitions.  For 
example,  program  No.  13  which  is  the  largest  program,  showed  counts  of  185 
and  746  for  operations  and  operands,  respectively,  under  the  first  interpreta¬ 
tion,  and  counts  of  118  and  900  for  the  second  interpretation.  The  effects 
on  the  metrics  under  the  two  interpretations  are: 


estimated  length 

9,645 

vol ume 

91 ,902 

level 

0.00365 

minimum  volume 

334.6 

effort 

25.243 

global  level 

1.1037 

versus  8,512  (11.7%  smaller) 
82,373  (10.3%  smaller) 
0.00212  (41.7%  smaller) 
174.2  (47.9%  smaller) 
38.945  (54.2%  larger) 
0.607  (45%  smaller) 


The  variation  in  these  metrics  is  indicative  of  the  effect  that  the  sub¬ 
jective  choices  (8  different  types)  can  cause.  In  a  separate  comparison,  the 
single  metric,  effort,  for  the  8  options  (for  program  No.  1)  were:  0.783, 
0.881,  0.937,  1.010,  1.065,  0.764,  0.794,  0.679.  This  variability,  which  is 
over  50%  (from  min  to  max)  is  evidence  of  a  lack  of  "objectivity"  in  this 
(and  other)  measures. 

2. 2. 1.1  Complexity  (Software  Science  Interpretation) 

In  the  abstract  of  the  paper  by  Fitzsimmons  and  Love  it  is  noted  that 
complexity  of  programs  can  be  measured  by  the  theory  of  Software  Science. 

It  was  difficult  to  locate  precisely  where  in  the  paper  this  complexity  is 
measured  because  the  word  appears  only  incidentally  in  the  text.  It  was 
determined,  by  direct  inquiry,  that  it  was  measured  by  the  effort  metric. 
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This  use  of  an  extensive  measure  for  complexity  is  indeed  novel  and  does 
not  correspond  to  intuition  or  to  any  other  measures  advanced  by  others. 
Halstead  states  that  complexity  of  a  program  is  measured  by  the  total  number 
of  elementary  discriminations  required  to  produce  it,  and  this  count  depends 
on  the  bulk  of  the  program  more  than  on  its  logical  structure. 

The  previously  published  measures  of  complexity  had  to  do  with  intensive 
measures  such  as  the  (normalized-to-unity)  spectrum  of  the  program  listing 
across  its  indenture  levels,  or  the  density  of  branching  statements. 

The  recently  described  measure  of  complexity  by  T.  J.  McCabe 
(Reference  16)  is  the  cyclomatic  number  obtained  from  the  flow  graph  of  the 
program.  This  metric  is  described  in  a  later  section  when  the  topic  of  com¬ 
plexity  is  re-examined.  Suffice  it  to  say,  it  is  more  an  intensive  measure 
than  an  extensive  measure,  and  as  McCabe  points  out  (op.cit.)  it  is  easy  to 
write  a  program  that  is  physically  small  but  ultra  complex. 

The  complexity  measure  of  software  science  is  directly  related  to  the 
length  of  the  program  (the  total  number  of  operators  and  operands)  and  is 
finally  developed  on  an  absolute  basis  by  the  use  of  the  so-called  Stroud 
number,  which  is  taken  by  M.  Halstead  to  be  18  mental  discriminations  per 
second. 

This  Stroud  number  has,  as  its  basis,  some  physical  measurements  of  a 
human's  ability  to  discriminate  the  frames  of  a  kaleidoscopically  presented 
visual  sequence  of  images  (related  to  the  "flicker  rate"  in  motion  pictures). 
The  use  of  this  visual  discrimination  rate,  as  equal  in  value  to  the  mental 
discriminations  rate,  is  surely  questionnable. 

2.2.2  Software  Metrics 

Probably  the  best  starting  point  for  this  discussion  is  a  review  of  the 
metrics  presented  in  the  book  by  Tom  Gilb  (Reference  17).  This  serves  more 
to  cover  the  field  than  to  make  precise  the  concepts  and  definitions  of  the 
many  metrics  identified.  Following  this  is  a  list  of  metrics  having  a 
reasonable  likelihood  of  surviving  through  test  and  time. 
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2. 2. 2.1  Review  of  Gilb  Metrics 


Maintainability 

The  first  definition  offered  by  Gilb  is  that  of  maintainability.  He 
defines  it  as 

"the  probability  that,  when  maintenance  action  is  initiated  under 

stated  conditions,  a  failed  system  will  be  restored  to  the  condition 

within  a  specified  time." 

That  definition  is  essentially  the  same  as  that  used  for  hardware.  In 
the  hardware  case  the  measure  is  almost  always  applied  in  a  bottoms-up  way, 
that  is  the  maintainability  is  derived  for  each  major  assemblage  from  the 
records  of  its  contained  minor  assemblies;  the  system’s  figure  is  derived 
from  the  major  assemblies.  Work  records  on  the  times  to  fix  are  estimated 
during  design,  and,  once  hardware  is  delivered,  records  are  kept  of  the 
actual  fix  times. 

Software  should  be  amenable  to  the  same  broad  guidelines.  Some  modules 
are  likely  to  be  more  easily  fixed  than  others  and  a  better  systems-wise 
figure  can  be  developed  from  the  bottoms-up  composition.  The  records  of 
maintenance  of  individual  modules  should  be  used  to  extrapolate  for  new 
errors.  The  fact  that  the  process  of  error-finding  tends  to  have  long  periods 
between  finds  (probably)  does  not  alter  the  fundamental  measure  of  the 
average  time  to  fix.  This  is  (probably)  so  because  the  late  occurring 
errors  are  (probably)  not  of  a  different  level  of  difficulty  than  the  early 
occurring  errors.  (Should  there  be  a  trend  towards  longer  fix  times  with  the 
"age"  of  the  error,  a  model  would  need  to  be  developed). 

Logical  Complexity 

Gilb's  introduction  into  this  topic  identifies  early  work  by  L.  Farr  and 
H.  J.  Zagorski,  who  used  the  IF  statement  density  as  a  measure  of  the  logical 
complexity.  Gilb  also  mentions  "psychological"  (his  quotes)  complexity  of 
source  programs  and  refers  to  some  statistical  work  by  L.  M.  Weissman  which 
correlated  metrizable  program  aids  (comments,  indentations,  etc.)  to 
productivity  and  accuracy. 
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Structuredness 

One  of  the  metrics  identified  by  Gilb  is  structuredness.  This  was  one  of 
many  metrics  proposed  by  TRW  in  a  study  for  the  National  Bureau  of  Standards. 

Structuredness  is  one  of  12  low-level  metrics  identified  by  Gilb,  the 
others  are:  device  independence,  completeness,  accuracy,  consistency,  device 
efficiency,  accessibility,  communicativeness,  self-descriptiveness, 
conciseness,  legibility,  and  augmentability. 

for  structuredness,  there  are  9  submetrics  which  are,  in  actuality, 
questions  concerning  the  existence  of  module  size  limits,  program  flow,  and 
so  forth.  Gilb's  Figure  51  (page  103)  can  be  referred  to  for  identification 
of  the  particular  questions.  It  does  not  appear  that  the  underlying  metrics 
have  any  quantitative  basis,  and  necessarily  have  either  a  zero  or  an  all 
value. 

Typical  of  a  question,  under  a  column  headed  "Definition  of  Metrics  to 
Measure  Structuredness,"  is:  "Do  all  subprograms  and  functions  have  only 
one  entry  point?"  Here,  should  the  answer  be  no,  there  is  no  way  of 
differentiating  between  "al 1 -but-one"  and  "none." 

Presumably  a  yes  answer  to  all  questions  would  indicate  a  perfectly 
structured  program.  Using  these  the  characterizing  features  (from  Figure  51 
of  Gilb)  the  program  would  be  one  which: 

A.  Has  rules  for  transfer  of  control  between  modules. 

B.  Has  limited  modules  sizes  (Note:  the  limit  is  not  specified). 

C.  Has  the  ordering:  commentary  header  block,  specification  statements, 
executable  code  (Note:  it  is  hard  to  imagine  a  program  that  does  not  follow 
the  order). 

D.  Subprograms  all  contain,  at  most,  one  point  of  exit. 

E.  Subprograms  and  functions  all  have  only  one  entry  point. 

F.  Program  flow  is  always  forward,  except  where  commented. 

G.  Overlay  structure  is  consistent  with  the  subprogram's  sequencing. 

H.  Is  subdivided  into  modules  in  accordance  with  readily  recognized 
functions . 

I.  Is  written  in  standard  constructs. 


o/ 

oot ;oi4>y 


17 


These  submetrics  are  then  scored  as  to  their  "correlation"  with  a  "high 
score  for  the  metric."  The  use  of  "correlation"  as  a  descriptor  for  sub¬ 
jective  judgment  is  highly  questionable:  there  are  no  numbers  to  associate 
with  the  identified  metrics,  and  the  numbers  associated  with  the  "score,"  if 
present  at  all,  are  certainly  vague. 

Nonetheless,  the  "quantifiability"  of  the  metrics  is  judged  against  six 
categories  which,  while  neither  exhaustive  nor  mutually  exclusive,  are 
nonetheless  indicated  as  such  by  the  tabular  entries. 

The  other  12  metrics  are  probably  treated  in  the  same  way  as  the 
structuredness  metric,  and,  beyond  their  identification,  do  not  appear  to 
merit  additional  inquiry.  (Self-descriptiveness,  communicativeness,  and 
accessibility,  for  example,  appear  to  be  invented  to  exercise  the  invention 
process,  and  do  not  represent  useful  metrics;  others,  such  as  augmentabi 1 i ty , 
may  have  some  value). 

Reliability 

Gilb's  definition  of  system  reliability  is  in  close  accord  with  the 
customary  (hardware)  definition.  It  states  that  "reliability  is  the  proba¬ 
bility  that  the  system  will  perform  satisfactorily  (with  no  malfunctions)  for 
at  least  a  given  time  interval,  when  used  under  started  conditions."  This 
is  modified  only  slightly  under  the  definition  offered  later.  Gilb's  variation 
of  his  definition  for  system  reliability  when  applied  to  program  or  software 
reliability  are  minor,  a  particular  machine  is  denoted,  and  operations  are 
"within  design  limits." 

Repairabil ity 

The  concept  of  repai rabi 1 i ty  is  a  variation  of  the  maintainability  con¬ 
cept.  The  emphasis  is  on  the  probability  of  a  repair  within  a  specified  time, 
when  maintenance  is  performed  under  specified  conditions.  The  requisite 
tools,  parts,  and  men,  are  assumed  to  be  available  at  the  start,  and  this  is 
one  of  the  specified  conditions. 
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Serviceabi  1 ity 

This  metric  is  taken  from  hardware  reliability  and  is  the  degree  of  ease 
or  difficulty  with  which  a  system  can  be  repaired.  It  is  not  considered 
quantifiable  at  present. 

Availability 

Again,  from  the  hardware  reliability  definition,  this  is  the  probability 
a  system  is  operating  satisfactorily  at  any  point  in  time.  It  is  usually 
measured  by  a  ratio  of  times  or  mean  times,  and  Gilb  offers  three  variations 
of  the  concept  (intrinsic-,  operational-,  and  use-availability). 

Attack  Probability 

This  metric  is  one  of  several  Gilb  suggests  in  the  security  aspects  of 
programs.  This  metric  is  the  probability  of  an  attack  (of  a  particular  type) 
on  a  system  during  a  particular  time  interval. 

These  attacks  can  be  considered  to  be  active  (malicious)  or  passive 
(typified  by  invalid  data). 

Security  Probability 

This  is  described  by  its  alternative  title,  attack  repulsion  probability, 
and  is  a  metric  gauging  the  probability  of  a  successful  rejection  in  the 
system  at  any  time.  The  attack  type  is  specified.  Gilb  states  that  this 
concept  is  close  to  the  concept  of  error  detection  probability.  This  is  less 
true  of  active  attacks  (which  may  not  persist)  than  it  is  for  passive 
attacks  such  as  bad  data. 

Integrity  Probability 

This  probability  is  the  probability  of  no  successful  attack  on  the 
system: 

■  MyO-S)] 

where  At  is  the  attack  probability  for  a  particular  time  interval,  and  S  is 
the  probability  of  rejection  (for  all  times). 
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Accuracy 

Several  examples  of  the  metric  and  a  discussion  contrasting  it  with 
precision  are  given  by  Gilb.  The  measurement  ratio,  correct  data/all  data, 
appears  to  be  too  vague  for  use  involving,  as  it  does,  the  idea  of  "correct 
data."  Usually  accuracy  involves  a  continuum  of  values  so  that  "correct" 
data  is  too  narrowly  defined  for  practical  usage. 

Precision 

The  suggested  measure  of  this  metric,  which  aims  to  gauge  the  degree 
"to  which  errors  tend  to  have  the  same  root  cause,"  is  the  ratio  formed  by 
dividing  the  number  of  actual  errors  at  source,  by  the  number  of 
corresponding  root  errors  observed  in  total  caused  by  source  bugs. 

The  difficulties  in  first  knowing  how  many  errors  there  are  at  the  source 
seem  unsurmountable,  and  tying  together  the  "corresponding"  errors  with  the 
source  would  not  seem  to  be  an  easy  task. 

Error  Detection  Probability 

Gilb  suggests  a  categorization  of  the  error  types  and  an  assignment  of 
the  likelihood  of  detection  of  errors  of  the  pre-specified  type.  The  failure 
to  include  time  aspects  into  the  problem  makes  for  a  flawed  definition.  The 
probability  of  an  eventual  detection  of  an  error  is  (probably)  unity  for 
almost  all  error  types. 

Error  Correction  Probability 

As  defined  by  Gilb,  this  is  the  probability  of  reconstructing  "data  in 
the  form  and  content  originally  intended."  This  is  a  vague  concept  when 
identification  of  the  random  event  is  sought.  The  originally  intended  form 
and  content  is  generally  not  known,  rather  it  develops  as  effects  are  judged 
unsatisfactory  and  tentative  changes  are  made.  There  is  a  chance  that  the 
repair  made  will  have  an  error  that  may  lie  undiscovered  for  a  period  of 
time,  and  so  time  should  be  involved  in  the  measure  in  some  way. 

Logical  Complexity 

In  the  text  two  metrics  for  logical  complexity  are  identified,  the 
numoer  of  binary  decisions  and  the  ratio  of  absolute  logical  complexity  to 
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total  complexity.  But  Gilb  also  suggests  under  the  Figure  83  on  page  161 
that  it  be  measured  by  the  number  of  possible  logical  path  combinations  in 
a  program. 

In  this  respect  Gilb  illustrates  with  an  unanswered  question,  the  defect 
in  using  even  the  density  of  branching  statements  as  a  measure  of  complexity. 
In  his  Figure  84,  two  programs  are  shown,  one  which  has  6  binary  decision 
points  and  the  other  only  one.  But  for  a  sufficiently  large  number  of  total 
instructions  (say  239  as  indicated  in  the  description)  the  density  of  the 
clearly  more  complex  program  is  less  than  the  ultra-simple  one.  This  alert 
is  examined  in  the  discussion  of  complexity  later  in  this  report. 

Flexibility 

Gilb  defines  this  as  that  part  of  complexity  that  is  useful,  and  it  is 
the  ratio  of  useful  to  total  that  is  the  metric. 

Built-in  Flexibility 

This  is  defined  as  the  ability  of  a  program  to  immediately  handle 
different  logical  situations.  It  must  be  built-in  in  order  to  respond 
without  loss  of  time. 

Adaptability  (open-ended  flexibility) 

Gilb  acknowledges  the  difficulty  of  originating  a  metric  for  this  con¬ 
cept  and  suggests,  as  a  tentative  measure,  the  count  of  the  linkages  between 
modules.  This  is  the  same  as  the  metric  used  later  for  structural 
complexity. 

Tolerance 

This  is  defined  as  the  ability  of  the  system  to  accept  different  forms 
of  the  same  information  as  valid.  The  proposed  metric  is  the  count  of  the 
number  of  different  variations  that  can  be  handled  by  the  system,  where 
variation  means  the  different  media,  different  formats  of  input,  or  logical 
variations  (such  as  misspellings  and  synonyms). 
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Generality 

The  "degree  of  applicability  of  a  system  within  a  stated  environment" 
constitutes  generality.  Its  measurement  is  subjectively  assigned  (0  to  1). 

Portability 

This  is  defined  as  the  ease  of  conversion  of  a  system  from  one  environ¬ 
ment  to  another.  The  metric  is  obtained  by  first  forming  the  ratio  of  the 
resources  required  to  move  the  program  to  a  target  environment  to  the 
resources  needed  to  create  the  program  for  the  target  environment,  and  then 
subtracting  the  ratio  from  unity.  The  result  is  the  ratio  of  the  cost  dif¬ 
ference  to  the  creation  cost  and,  on  the  extremes,  agrees  with  an  economic 
measure  of  portability,  because  for  a  zero-cost  move  the  portability  is 
unity,  and  for  a  cost  equal  to  the  creation  cost,  the  portability  value  is 
zero. 

Compatibility 

This  attribute  is,  according  to  Gilb,  related  to  the  concept  of  porta¬ 
bility,  the  difference  being  that  portability  is  a  characteristic  of  a  single 
system  whereas  compatibility  applies  to  an  average  over  a  class  of  systems. 
This  distinction  provides  the  metric,  an  average  portability  over  the 
collection  of  program  systems. 

Redundancy  Ratio 

This  is  the  first  of  what  are  called  structural  metrics  by  Gilb.  This 
ratio  generally  is  formed  by  taking  the  actual  count  of  quantities  to  the 
minimum  possible  count. 

Hierarchy 

This  structural  metric  describes  the  number  of  indenture  levels  and  the 
spectrum  of  program  elements  across  these  levels. 

Structural  Complexity 

As  noted  earlier  in  the  section  concerning  adaptability,  this  is 
measured  by  the  number  of  modules  (absolute)  or  the  ratio  of  linkages  to  the 
total  number  of  modules.  This  is  an  easy  metric  to  derive  for  some  languages 
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as  Gilb  shows.  For  FORTRAN  the  modules  are  counted  by  the  number  of 
subroutines  and  functions,  and  the  number  of  linkages  is  the  total  of 
subroutine  parameters  and  the  references  to  the  common  area. 

Modularity 

Although  modularity  is  stated  to  be  a  synonym  for  structural  complexity, 
it  seems  to  stress  the  number  of  modules  and  not  the  linkages. 


Distinctness 

Distinctness  as  defined  by  Gilb  involves  errors  and,  in  fact,  is 
measured  by  a  ratio  between  the  number  of  bugs  in  the  module  and  the  number 
that  are  common  to  the  module  and  another  ("simultaneously").  It  is  hard  to 
see  how  this  ties  to  the  intuitive  concept  of  uniqueness,  particularly  how 
errors  are  necessary  components  of  distinctness. 

Effectiveness 

Among  the  performance  metrics,  effectiveness  is  listed  first.  It  is  a 
probability  of  "success"  within  a  given  time  and  specified  environment.  The 
"success"  means  meeting  an  operational  demand.  Gilb  composed  efficiency 
from  three  probabilities:  reliability,  readiness  probability,  and  design 
adequacy  (on  a  scale  from  0  to  1). 

Efficiency 

This  attribute  is  defined  as  the  ratio  of  useful  work  to  the  total 
expended. 

Cost 

Among  financial  metrics  are  costs  and  its  major  subdivisions,  fixed  and 
variable.  Gilb  uses  the  terms  capital  and  operational. 

Time 

Computer  and  "Human"  time  resource  metrics. 


Space  Metrics 

This  is  more  commonly  called  the  size  of  a  program.  It  can  be  measured 
on  an  atomic  level  by  bits  and  bytes  and,  on  the  more  common  scale,  by  the 
number  of  instructions. 
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Information 

Gilb  says  that  information  content  of  a  program  is  not  directly 
measurable,  and  suggests  use  of  "useful  data"  as  an  indirect  means  for 
measurement. 

Evolution 

This  is  a  measure  of  the  incremental  change  to  a  system  during  a  time 
interval,  t.  If  the  change  is  so  pervasive  that  it  constitutes  a 
substitution,  the  metric  would  have  a  value  of  unit. 

Stability 

Stability  is  the  complement  of  Evolution  and  it  denotes  the  percentage 
of  unchanged  content  of  a  program  (over  a  specified  time  period). 

2.2.3  Candidate  Metrics 

Clearly  some  of  the  questions  that  should  have  been  asked  of  the 
community  several  years  ago  are: 

A.  Are  any  attributes  worth  study? 

B.  Which  attributes  are  useful? 

C.  Can  these  be  measured  in  a  form  useful  to  the  community? 

It  is  clear  from  inspection  of  the  Gilb  metrics  that  there  are  many  that 
will  not  survive  the  tests  required  of  practical  gauges.  Most  of  the  13 
low-level  metrics  identified  by  Gilb  have  little  hope  of  common  usage.  The 
discussion  concerning  structuredness ,  in  that  section,  indicates  that  the 
concept  is  initially  vague  and  becomes  amorphous  after  its  component  parts 
are  identified  (in  the  form  of  questions). 

Of  the  metrics  listed  above,  the  following  are  considered  of  primary 
value:  reliability,  complexity,  cost  and  time. 

Regarded  as  secondary  in  importance  are:  maintainability  and 
avai labi 1 i ty . 
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Supplementing  these  metrics  are  some  that  history  may  judge  to  be  of  more 
value  than  any  of  the  metrics  identified  above:  mean-time-to-next  error, 
mean-time-to-perfection,  error  content,  testedness,  and  purification  level. 

Excepting  the  complexity  metric,  it  does  not  seem  necessary  to  amplify 
on  the  previous  Gilb  definitions,  and  the  following  subsections  deal  with  the 
augmenting  metrics. 

2.2.3. 1  Mean-Time-To-Next-Error 

Of  primary  importance  in  the  testing  of  programs  is  the  decision  on 
whether  or  not  to  release  a  given  module  or  program.  A  good  guide  to  this 
choice  lies  in  the  time  pattern  of  the  errors  found,  whether  this  pattern  lies 
in  a  data  base  metered  by  CPU  units,  hours,  days,  or  weeks  is  not  relevant 
(except  as  its  potential  future  uses  may  have  to  be  considered).  If  the  time 
pattern  indicates  a  steady  state  or  constant  error  rate,  or,  even  worse, 
shows  an  increasing  failure  rate,  there  is  clearly  no  reason  for  releasing 
the  module  and  much  evidence  to  the  contrary.  Once  a  pattern  of  decreasing 
counts  (per  unit  time)  is  achieved,  any  of  several  models  can  be  applied  to 
the  data  that  the  error  pattern  represents,  and  estimates  of  the  mean-time- 
to-next-error  can  be  obtained. 

It  is  the  magnitude  of  this  mean-time-to-next-error,  or  more  commonly 
called  the  mean-time-to-fai lure,  MTTF  (which  for  a  certain  probability 
distribution,  and  steady  state  conditions,  is  the  same  as  the  mean-time- 
between-fai lures  [MTBF]),  considered  in  the  context  of  its  expected  use,  that 
is  important.  For  real-time  systems,  governing,  for  example,  weapons  or 
aircraft,  the  MTTF  should  be  several  times  as  large  as  the  mission  duration. 
The  proper  figure  for  the  MTTF  is  determined  by  the  reliability  specified  by 
the  customer  for  the  system. 

Values  for  the  MTTF  are  available  in  any  of  several  models:  Jelinski- 
Moranda,  Shooman,  Schick-Wol verton,  Moranda  Geometric  Purification,  Moranda 
Hybrid  Geometric-Poisson. 

Littlewood  and  Verrall  (Reference  18)  avoid  MTTF  and  insist  instead  on 
percentiles  (such  as  the  median)  of  the  distribution  describing  the  time 
between  errors. 

r 
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It  is  important  to  note,  that  for  all  models  the  MTTF  is  a  parameter 
that  is  changed  by  either  event  or  time.  The  Jel inski -Moranda  model  has  an 
MTTF,  indexed  by  the  dummy  variable  i,  which  increases  at  the  occurrence  of 
each  error,  and  can  be  given  in  terms  of  the  model  parameters  N  and  <j>  as 

NTTFJ-M  =  [N-(i-l  )]<P 

The  Schick-Wol verton  model  has  an  "instantaneous"  MTTF  depending  on  both 
time  and  event,  and  it  has  an  averaged  MTTF  that  is  obtained  from  the  first 
moment  of  the  Rayleigh  distribution  for  the  time  of  next  error.  Thus, 


MTTFS-W 


r  i  i1/2 

_(N-n)<(>  J 


For  the  Geometric  Purification  model,  the  MTTF  is 


MTTF 


G-P 


where  D  is  the  failure  rate  for  the  first  error,  k  is  the  geometric  ratio 
which  is  used  to  obtain  the  error  rates,  and  n  is  the  number  of  found  errors. 


The  Shooman  MTTF  is  given  by 


MTTFs  '  [c  ErEc(tE  ■’ 

where  C  is  a  proportionality  constant,  Ej  is  the  total  error  content,  and  Ec(t) 
is  the  total  number  of  errors  found. 

2.2. 3.2  Mean-Time-To-Perfection 

Some  models  permit  an  estimate  of  the  mean  time  required  to  achieve  an 
error-free  program.  Generally  this  estimate  is  accompanied  by  a  variance 
(standard  deviation)  that  is  so  large  that  it  has  little  or  marginal  utility. 

It  is  nonetheless  a  guide  to  management  and  it  is  changed,  and  generally 
made  more  precise,  as  more  errors  are  discovered. 
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The  simplest  way  to  form  this  estimate  is  to  sum  the  estimated  MTTF's 
for  all  remaining  errors;  hence  (using  MTTP  for  mean-time-to-perfection) ,  the 
estimate  so  formed  for  the  Jel inski -Moranda  model  is: 


1  1 
"ttpo-m  ■  1  2  iH 
j=n 

For  the  Schick-Wol verton  Model, 


MTTP 


S-W 


N-l  / — 

2  /t 

J=n 


1/2 


1 


(N-j  )4> 


This  last  formula  is  incorrectly  given  in  the  latest  Schick-Wol verton  paper 
(Reference  19). 


The  Shooman  model  does  not  permit  an  estimate  of  the  MTTP  because  the 
failure  rate  of  that  model  is  a  continuous  exponential.  The  mean  time  to 
achieve  a  zero  with  the  exponentially  decreasing  failure  rate  is  infinite. 

The  Moranda  Geometric  Purification  model  also  does  not  have  a  finite 
average  time  to  perfection.  Even  though  discrete,  the  failure  rate  does  not 
attain  a  zero  value. 


The  Littlewood  and  Verrall  model,  based  on  Bayesian  adjustment,  does  not 
involve  a  parameter  that  can  be  directly  related  to  the  MTTF,  and  it  is 
required  that  some  alternative  be  found.  This  can  be  provided  by  any  of  the 
percentiles  of  the  distribution  formed  by  convolution  of  the  family  of 
related  exponential  distributions  they  use  in  their  examples.  It  is  neces¬ 
sary,  however,  to  rely  on  the  most  recently  available,  "a  posteriori"  distri¬ 
bution  for  one  of  the  two  parameters,  and  to  continue  the  assumption  con¬ 
cerning  the  way  that  the  sequence  of  values  for  the  other  parameters  are 
related.  It  presents  a  difficult  problem  analytically  and  probably  has 
practical  objections. 
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The  recent  publication  by  A.  L.  Goel  and  K.  Okumoto  (Reference  20)  has 
relevance  to  this  and  some  of  the  other  problems.  Their  work  in  the  present 
context  employs  a  family  of  distributions  that  are  the  same  as  those  used  by 
Jelinski  and  Moranda  but  with  an  essential  difference,  they  assume  an 
imperfect  repair  and  account  for  it  with  a  parameter,  p,  that  is  the  same  for 
all  errors.  Using  these  variations,  the  distributions  of  the  "first  passage" 
times  (zero  errors)  and  of  the  times  to  achieve  various  levels  of  purification 
are  derived. 

2. 2. 3. 3  Error  Content 

Three  models  can  be  used  to  derive  estimates  of  the  error  count.  The 
Jel inski -Moranda  model  accomplishes  it  through  use  of  equations  developed 
from  the  assumption  that  there  is  a  direct  proportion  between  error  content 
and  failure  rate.  The  corresponding  Shick  and  Wolverton  assumption  is  that 
the  failure  rate  is  proportional  to  both  the  number  of  errors  and  the 
"debugging  time."  The  Shooman  model  can  be  used  at  two  or  more  separated 
time  intervals  to  estimate  the  error  content.  From  observations  of  the 
average  MTTF  for  these  intervals,  parameters  of  the  linear  relation  between 
failure  rate  and  error  content  can  be  found  by  simultaneous  equations  (for 
two  intervals)  or  by  least  squares  (for  three  or  more). 

2. 2.3.4  Purification  Level 

Although  some  models  do  not  measure  error  content  and  may  not  achieve  a 
perfect  state,  there  is  a  measure  that,  in  some  cases,  can  be  used  to  describe 
the  state  of  perfection  achieved  at  a  given  point  in  time.  For  error-content 
models,  the  ratio  of  the  number  found  to  the  total  number  (estimated)  provides 
the  reasonable  estimate.  For  the  Moranda  Geometric  Purification  process,  the 
purification  state  can  be  estimated  by  taking  the  ratio  of  the  initial 
failure  rate  to  the  achieved  failure  rate. 

The  purification  level  or  percentage  is  clearly  of  more  value  than  the 
error  content  since  the  absolute  number  is,  by  itself,  generally  a  poor 
indicator  of  status  because  it  is  size-of-program  related. 
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The  several  estimators  of  the  purification  percentage  are  (in  terms  of 
their  defined  parameters ): 


Jel inski -Moranda 

R  X  100 

Schick-Wolverton 

KT  X  10° 

N 

E 

Shooman 

^  x  100 
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Geometric 

Purification 

(1-k")  (100) 

2. 2. 3. 5  Testedness  (Degree  to  Which  a  Program  Has  Been  Tested) 

A  metric  of  a  different  kind  is  represented  by  the  degree  to  which  the 
program  has  been  tested.  There  are  several  different  types  of  "coverage"  for 
a  program,  where  "coverage"  means  that  the  program  "elements"  have  been 
executed. 

E.  C.  Miller  (Reference  21)  presented  a  useful  list  of  several  different 
coverage  types  in  a  sequence  reflecting  the  increasingly  larger  size  of  the 
covering  unit.  The  lowest  level  of  coverage  is  obtained  when  every  statement 
is  executed  at  least  once,  the  next  level  is  achieved  when  each  segment 
associated  with  the  explicit  or  implicit  predicate  outcome  is  executed. 

For  complex  programs  involving  nested  loops,  the  test  coverage  may  neces¬ 
sarily  be  limited  to  the  exercising  of  the  program  so  as  to  test,  one  time, 

\ 

all  so-called  boundaries  and  interiors  of  loops, \it  being  assumed  that  all 
segments  are  exercised.  (Boundaries  are  the  entries  and  exits  from  a  loop.) 

A  higher  order  of  coverage  consists  of  multiple  passes  through  loops,  these 
are  tests  that  iterate  all  loops  up  to  a  certain  specified  limit  (even  1), 
and  provides  additional  coverage.  The  ultimate  test  coverage  (with  several 
other  types  in  between)  exercises  all  logical  paths  through  a  program. 
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One  additional  type  is  afforded  by  the  testing  of  the  type  described  in 
the  earlier  work  (Reference  1),  where  tracks  were  identified.  Coverage  by 
tracks  is  intermediate  between  segment  coverage  and  logical  path  coverage. 

The  metric  for  any  of  the  types  is  simply  the  ratio  of  the  number  of 
"elements"  (defined  as  instructions,  segments,  branches,  predicates,  tracks 
or  logical  paths)  to  the  total  number  of  these  elements. 

A  new  concept  was  introduced  by  Moranda  (Reference  1)  where  the  diffi¬ 
culty  of  enumerating  the  number  of  different  elements  is  avoided.  By  using 
random  numbers,  it  is  possible  (under  some  assumptions  that  are  reasonable 
or  acceptable  to  some  and  questionable  or  unacceptable  to  others)  to  esti¬ 
mate  the  total  number  of  program  elements  that  will  eventually  be  achieved 
by  random  numbers.  This  technique  was  used  in  the  original  work 
(Reference  1)  to  estimate  the  total  number  of  "tracks,"  but  could  as  easily 
be  used  to  derive  the  asymptotic  limit  to  the  number  of  logical  paths. 

2. 2. 3. 6  Complexity 

As  noted  in  the  discussion  on  the  effort  metric  employed  for  complexity- 
measures  by  the  Software  Science  advocates,  the  use  of  an  extensive  measure 
for  complexity  runs  counter  to  intuition.  The  total  number  of  "elementary 
discriminations"  required  to  produce  a  program  does  not  seem  to  properly 
reflect  the  structural  aspects  of  complexity,  for  a  "straight-line"  program 
(no  loops)  of  extreme  length  would  have  a  high  effort  value,  but  might  be 
judged  rather  simple. 

Other  measures  were  suggested  in  that  same  discussion.  The  density  of 
branching  statements  was  suggested,  but  as  noted  by  McCabe  in  Reference  16, 
the  density,  as  measured  by  instruction  count,  may  be  misleading.  In  that 
reference,  a  reasonably  complex  program  containing  six  branches  had  so  many 
(hypothesized)  instructions  that  the  density  of  branching  statements  was 
less  than  a  short  straight-line  program  (with  one  branch). 

It  is  clearly  necessary  to  alter  the  concept,  and  base  the  metric  on  the 
segment  counts,  rather  than  the  instruction  counts.  This  is  a  reasonable 
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position  to  take  because  some  segments  may  contain  a  very  large  number  of 
instructions.  As  far  as  the  intricacies  or  complexities  of  a  program  are 
concerned,  all  segments  are  the  same  and  do  not  depend  on  the  number  they  are 
comprised  of. 

Thus,  a  more  satisfactory  metric  for  complexity  would  be  either  the 
number  of  segments  or  the  number  of  logical  paths.  Since  the  latter  are 
difficult  to  count  in  many  cases,  the  former  can  be  used,  even  though  the 
way  they  connect  is  not  measured  thereby. 

Another  measure  of  complexity  which  may  be  of  use  is  the  indenture  level 
spectrum.  This  concept  is  rather  simple  in  that  it  tallies  into  each  inden¬ 
ture  level,  each  instruction  of  the  program.  By  dividing  the  number  in  each 
category  by  the  total  number  of  instructions,  a  normal i zed- to-unity  spectrum 
can  be  produced.  There  are  clearly  some  deficiencies  in  this  approach  since 
a  program  that  "shifts"  back  and  forth  between  two  adjacent  levels  is  not 
judged  to  be  more  complex  than  one  that  has  the  same  number  of  instructions 
at  each  level  and  "shifts"  but  once.  The  metric  would  require  a  complementary 
measure  to  provide  a  total  measure  of  complexity. 

A  far  better  metric  for  complexity  has  been  developed  by  T.  J.  McCabe 
(Reference  16).  He  suggests  that  the  program  be  represented  by  a  directed 
graph,  G,  in  the  usual  way.  The  way  the  nodes  (or  vertices)  and  segments 
(edges)  of  G  are  connected  is  measured  by  a  cyclomatic  number,  denoted  by 
V(G),  determined  by  the  number  of  edges,  vertices,  and  connected  components 
(where  the  latter  is  a  subgraph  of  G). 

McCabe  proves  a  theorem  that  permits  an  alternative  way  of  finding  the 
cyclomatic  number:  for  strongly  connected  graphs,  the  cyclomatic  number  is 
the  maximum  number  of  linearly  independent  circuits.  In  order  to  apply  this 
theorem,  it  is  necessary  to  form  a  strongly  connected  graph  by  looping  back 
from  the  exit  node  to  the  entrance  node. 

It  is  generally  easy  to  identify  the  cyclomatic  number  of  most  reasonably 
wel 1 -structured  programs  of  small  to  moderate  size.  Where  the  program  is 
extensive,  the  algebra  set  up  by  McCabe  can  be  used  to  calculate  the  number. 
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Section  3 

COVERAGE  BY  RANDOM  AND  CONSTRUCTED  CASES 


3.1  INTRODUCTION  AND  BACKGROUND 

The  following  introductory  section  is  essentially  a  duplicate  of  the 
text  material  used  to  introduce  the  topic  of  coverage  in  Reference  1. 

3.1.1  Framework  for  Representation 

In  the  customary  renditions  of  program  flowcharts,  each  (rectangular) 
block  represents  either  a  simple  instruction,  or  a  group  of  operations,  with 
a  single  output,  while  each  diamond  represents  a  single,  explicit  or  implied, 
predicate  which  has  two  or  more  exit  options.  Connecting  the  blocks  and 
diamonds  of  a  flowchart,  are  directed  lines  denoted,  and  referred  to,  as 
arrows.  These  lines  represent  the  options  possible  and  are  called  flow-of- 
control  arrows.  These  fundamental  building  blocks  are  adequate  for  the 
static  or  structural  description  of  a  program,  but  are  not  convenient  for 
representing  its  operational  aspects.  The  basic  operations  are  better 
defined  in  terms  of  some  simple  program  components.  These  lend  themselves 
to  mathematical  descriptions  and  they  motivate  the  choice  for  the  "atomic" 
or  fundamental  unit  of  description. 

First,  it  is  noted  that  an  instruction  in  a  program,  while  easy  to  define 
(statically)  in  "machine  language",  becomes  rather  difficult  in  most  of  the 
higher  order  languages.  Thus  a  "clear  and  add"  instruction,  in  machine 
language,  causes  a  register  (accumulator)  to  be  set  to  zero  and  another 
register  to  be  transferred  to  the  cleared  register  and  nothing  more.  Once 
the  final  bit  is  transferred,  the  machine  waits  until  the  next  instruction, 
which  is  generally  started  by  a  timing  or  clock  pulse.  On  the  other  hand, 
the  concept  of  an  instruction  in  the  higher  languages  is  less  clear.  An 
"instruction"  in  ALGOL,  for  example,  is  either  a  statement  or  a  declaration, 
and  in  either  case  is  used  to  indicate  required  compiler  (as  against  computer) 
actions.  As  a  result  of  compiler  action,  an  object  program  with  computer 
interpretable  instructions,  is  produced. 
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Thus,  there  is  a  spectrum  of  statements  in  that  language:  the  simplest 
type  is  an  assignment,  such  as  X : = 1 ;  while  one  of  the  more  complex  statements 
is,  begin  ...  end,  which  groups  statements  together  to  form  compound 
statements  (and  blocks). 

In  any  higher  order  language  where  grouping  is  required,  there  is  a  need 
for  so-called  delimiters  (explicit  or  implicit)  which  can  be  used  as  bound¬ 
aries  for  the  steps,  and  form  the  building  blocks  of  a  program.  A  similar 
device  is  required  in  the  description  of  dynamic  operations  -  a  means  of 
grouping  instructions  into  fundamental  operational  units. 

Generally,  the  linking  of  instructions  can  be  represented  by  means  of  a 
Boolean  indication,  with  the  value  1  used  where  the  instructions  are  or  can 
be  "contiguous",  and  0  used  to  denote  the  fact  that  they  are  not  connected. 
These  Boolean  values  could  be  used  as  entries  of  a  connection  matrix  whose 
row  and  columns  are  numbered  to  accord  with  an  (arbitrary)  numbering  scheme 
for  the  steps.  But  a  straightforward  application  in  this  manner,  on  the 
instruction  level,  would  normally  produce  inordinately  large  and  unmanage¬ 
able  connection  matrices.  Some  of  the  redundant  information  in  such  a 
matrix  could  be  eliminated  if  certain  agreements  can  be  made:  for  example, 
if  step  1  is  always  followed  in  sequence  by  steps  2,  3,  and  4  and  there  is  not 
opportunity  for  branching  until  step  4  (at  least),  then  steps  1  through  4 
can  be  merged  or  combined,  and  three  of  the  rows  and  columns  of  the  connector 
matrix  could  be  eliminated.  This  reduction  in  redundancy  is  an  additional 
reason  for  choosing  groups  of  instructions  for  the  description. 

Because  certain  instructions  or  statements  have  more  than  one  output 
(such  as  if .. .then. . .else)  there  is  a  need  to  devise  a  convention  which  will 
permit  identification  of  each  of  the  exits.  If  statement  A  is  a  single¬ 
output  statement  and  it  connects  to  statement  B  which  has  multiple  outputs, 
the  notation  [A, 8),  which  is  "closed"  on  the  left  and  "open"  on  the  right,  is 
meant  to  imply  that  A  is  executed  and  control  is  passed  to  (or  toward)  B, 
but  that  B  is  not  executed,  but  it  is  next  in  line.  If  B  is  a  two-output 
instruction  and  connects  to  L]  and  L2,  then  both  [B,L-| )  and  [B,L2)  are  used 
to  describe  the  optional  branches  which  can  be  taken. 


MCDOWWm  004/0 14! 


34 


The  procedure  which  has  been  described  can  be  illustrated  by  a  flow  diagram 
of  a  very  simple  program.  In  Figure  1  is  a  combination  of  a  code  listing  on 
the  right  and  a  flow  diagram  on  the  left.  Numbers  refer  to  the  instructions 
listed.  The  program  is  designed  to  process  a  sequence  (one  or  more)  of  lists, 
with  each  list  consisting  of  "test  scores"  augmented  by  the  number  -1  (which 
is  not  a  test  score);  the  last  list  is  further  augmented  with  a  -2  (for  HALT 
purposes).  The  program  tallies  the  number  of  scores  within  each  list 
which  are  at  least  as  large  as  70  (passing),  and  also  tallies  the  total 
number  of  passing  scores  within  all  lists  (the  Grand  Sum). 

To  continue  with  the  description,  it  will  be  seen  in  Figure  1  that  the 
first  connection  to  a  branching  instruction  is  made  at  instruction  number  3. 

From  3  the  branch  taken  is  determined  by  the  predicate  (X=-2)  and  how  the 
input  to  3  (carried  out  of  2)  values  it  (true  or  false).  Thus,  instruction 
number  3  is  connected  to  14  and  to  4,  as  potential  (operating)  successors 
to  3.  In  the  same  way,  5  as  a  branching  statement  connects  to  6  and  10. 

A  variation  of  the  technique  which  is  usually  employed,  characterized 
by  connecting  "nodes"  (representing  sets  of  instructions)  is  proposed  here. 
Emphasis  in  this  variation  is  on  the  branches  which  emanate  or  terminate 
with  branching  instruction,  and,  in  fact,  the  fundamental  or  "atomic" 
element  in  the  representation  of  a  program  is  taken  to  be  a  segment  or 
string  of  instructions  between  two  branching  instructions.  More  precisely 
a  segment  is:  a  sequence  of  instructions  starting  with  either  a  START,  or 
a  branching  instruction,  and  ending  (but  not  inclusively)  with  the  first 
subsequent  branching  instruction,  or  a  HALT,  in  which  particular  case  the 
segment  is  considered  to  include  the  instruction  which  ends  it. 

As  an  example  of  the  way  segments  are  developed,  the  flow  diagram  in 
Figure  1  is  analyzed: 

51  =  [1,2,3) 

52  =  [3,14,15) 

53  =  [3,4,5) 

54  =  [5,10,11,12,13,3) 

55  =  [5,6) 

56  =  [6,8, 9, 5) 

S?  =  [6, 7,8, 9, 5) 


1 


1  GSUM  •  0 


2  READ  X 

3  IF  X* -2  GOTO  14 

4  SUM  -  0 

5  IF  X  =  -1  GOTO  10 

6  IF  X  <  70  GOTO  8 

7  SUM  +  SUM  -  1 

8  READ  X 

9  GOTO  5 

10  PRINT  SUM 

1 1  GSUM  +  GSUM  -  SUM 

12  READ  X 

13  GOTO  3 

14  PRINT  GSUM 

15  HALT 


Figure  V  Test  Scores  Program  and  How  Diagram 
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The  distinction  between  brackets  and  parentheses  is  important  and  has 
been  noted.  The  only  cases  where  square  brackets  are  used  on  the  right  are 
those  in  which  the  last  instruction  listed  is  a  HALT  (number  15  in  the  example). 

Any  particular  set  of  values  (for  the  coordinates)  of  the  input  vector 
(point  in  the  input  space),  causes  exactly  one  sequence  of  operations  to  be 
executed.  These  segments  linked  together  form  a  logical  path  through  the 
program.  This  is  also  called  an  execution  sequence. 

It  is  useful  to  modify  the  term  logical  path  with  the  work  realizable 
when  input  data  can  cause  it.  Before  data  is  entered,  possible  (or  feasible) 
logical  paths  can  be  formed  by  any  concatenation  of  contiguous  segments  which 
have  the  START-segment  first  and  end  with  a  HALT-segment.  In  the  case  a 
program  has  self-contiguous  segments  (loops)  or  one  or  more  concatenations 
which  join  end-to-end,  the  number  of  (possible)  repetitions  of  the  joined 
segments  is  arbitrarily  large  -  except  where  a  predetermined  number  of 
traversals  are  specified  in  the  program. 


The  following  sequences  of  segments  in  the  program  of  Figure  1  are 
illustrative  of  some  possible  or  feasible  logical  paths: 


S1S2 

S1S3S4S2 

S1S3S4S3S4S3S5S6S4S2 

S1S3S5S6S4S2 

S1S3S5S7S4S2 


The  first  path  is  of  minimum  possible  length,  linking,  as  it  does,  the 
START  -  and  HALT  -  segments.  The  last  two  are  interesting  in  that  they 
exhaust  the  collection  of  segments  (but  not  the  logical  paths). 


In  order  to  determine  realizable  logical  paths,  the  documentation  or 
"program  writeup"  must  be  considered.  In  this  simple  case  it  is  very  easy 
to  establish  data  which  will  realize  the  flows  represented  by  the  last  two 
sequences  of  the  above  list.  (It  should  be  noted  that  insofar  as  testing 
to  the  instruction- level  only  one  of  these  two  need  be  driven  but  to  obtain 
segment  or  branch-level  testing,  both  need  to  be  tested). 
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If  for  example  the  data  sequence  (stacked) 


x  =  35,  -1,  -2 

is  employed,  the  next  to  the  last  sequence  of  the  above  list  describes  the 
flow,  and  for  the  "stack" 

x  =  75,  -1,  -2 

the  last  sequence  describes  the  flow.  The  two  stacks  together  provide  an 
exhaustive  test  of  the  segments  of  the  program. 

Moreover,  a  single  sequence  35,  -1,  75,  -1,  ~2  would  also  produce  an 
exhaustive  test  of  the  segments  with  the  sequence  S-jS^S^-S^S^.  While 
these  do  not  exhaustively  test  the  realizable  logical  paths  (which,  without 
further  explicit  restrictions,  are  infinite  in  number),  it  is  well  to  note 
that  the  complete  segment-testing  partially  accomplishes  one  of  the  major 
purposes  of  case  selection,  that  of  exercising  all  instructions  so  as  to 
uncover  incompatibilities  with  the  machine  and  other  errors. 

This  limited  form  of  testing  brings  up  a  very  interesting  and  very 
obvious  observation  that  is  true  for  any  program  represented  as  a  collection 
of  segments:  if  a  program  consists  of  k  segments,  and  every  segment  can  be 
exercised  by  some  data  point,  then  only  k  data  points  are  required  to 
exhaustively  test  the  program  in  the  segment  testing  sense.  This  is  of  course 
very  useful  in  the  case  that  an  interactive  or  communicative  mode  of  testing 
is  employed. 

3.1.2  Extension  to  Testing  for  Track  Coverage 

Under  AFOSR  contract  AF  44620-74-C-0008,  MDAC  developed  a  model  which 
employs  random  numbers  as  input  data,  and,  on  the  basis  of  the  trial  numbers 
on  which  "new"  logical  paths  are  driven  by  the  input  data,  estimates  the 
asymptotic,  or  eventual,  level  of  testing  achieved  with  random  numbers. 

The  basic  analysis  mechanism  is  the  original  Jel inski -Moranda  model 
(Reference  22).  The  measurement  used  in  the  model  is  the  number  of  trials 
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occurring  between  the  discovery  of  new  logical  paths  (rather  than  times 
between  errors  which  comprised  the  raw  data  for  the  estimation  of  residual 
error  content  in  the  original  application  of  the  model). 

There  is  another  relevant  use  of  this  same  model.  If  the  probability 
law  governing  the  selection  of  input  data  is  known,  then  the  coupling  of 
information  derived  from  sampling  with  universal  (a  priori)  error  rate  data 
will  permit  an  estimate  of  the  operational  reliability  of  the  program.  This 
procedure,  also  developed  under  the  same  AFOSR  contract  was  reported  in 
Reference  23. 

A  second  model  employing  program  or  software  input  data  for  analysis,  is 
due  to  TRW  (Reference  24).  In  essence,  this  model  uses  a  subdivision  of  the 
input  data  space  into  equivalence  classes,  each  characterized  by  the 
particular  logical  path  exercised  by  all  of  its  members. 

This  subdivision  was  suggested  earlier  by  W.  Howden  (Reference  7)  and 
also  by  B.  Elspas,  et  al.,  (Reference  8).  In  applications  the  TRW  model  has 
been  used  in  the  estimation  of  software  reliability.  The  estimate  is  derived 
by  composing  the  assumed-to-be-known  probability  that  each  subdivision  is 
employed,  with  a  sample-derived  conditional  probability  of  committing  an 
error  when  the  subdivision  is  used.  The  problem  in  the  application  of  such 
a  model  is  the  difficulty  involved  in  the  formation  of  the  subdivisions, 
confirmed  by  almost  everyone  who  has  attempted  to  work  from  a  specified 
logical  path  to  the  descriptor  of  the  input  data  associated  with  it.  Another 
deficiency  occurs  when  the  model  is  used  for  estimation  because,  a  permanent 
program  is  assumed  which  does  not  change  to  remedy  the  found  errors.  The 
problem  of  precisely  carving  out  the  equivalent  classes  is  a  severe  barrier 
to  application  of  such  techniques.  It  is  probably  better  to  avoid  the  problem, 
as  done  in  the  application  of  random  numbers  described  in  Reference  1 ,  or  by 
using  techniques  like  those  described  by  W.  Miller  and  D.  Spooner  (Reference  3). 

The  use  of  random  numbers  as  inputs  to  a  software  package  has  fundamental 
limitations.  For  example  the  occurrence  of  an  input  which  takes  on  a  zero 
value  is  essentially  impossible  and  this  input,  and  others  of  a  similar 
nature,  must  be  supplied  to  produce  a  set  of  inputs  which  will  achieve  such 
values. 
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Nevertheless,  as  shown  in  earlier  work  (Reference  1),  the  fundamental 
limitation  can  be  numerically  estimated  for  a  given  program  on  the  basis  of 
the  set  of  logical  paths  effected  as  a  result  of  random  drivers.  It  can  be 
said  that  the  number  found  in  this  way  is  an  always  fair  and  often  an 
excellent  bound  on  the  total  number  of  logical  paths  which  are  ever  actually 
exercised. 

The  work  of  Miller  and  Spooner  avoid  these  problems  with  an  elegant 
substitute:  instead  of  attempting  to  solve,  in  the  input  data  space,  the  set 
of  equations  (or  inequalities)  associated  with  a  specified  logical  path,  they 
insert  a  new  set  of  variables,  one  at  each  branching  point  in  the  program. 

An  objective  function  of  these  variables  is  chosen  so  that  when  its  functional 
value  is  positive,  the  input  data  is  in  the  equivalence  set  associated  with 
the  specified  logical  path.  This  method  employs  standard  procedures  from 
the  field  of  system  optimization,  starting  with  a  randomly  chosen  initial 
point  in  the  input  domain. 

For  additional  background,  a  review  is  made  of  the  means  of  representing 

the  flow  graph  by  a  connection  matrix.  As  noted  in  prior  work  the  matrix  is 

constructed  by  assigning  a  1  or  0  as  an  entry,  according  to  whether  or  not 

there  is  a  connection  between  the  nodes  (or  segments)  corresponding  to  the 

associated  row  and  column  of  the  matrix.  A  simple  way  of  visualizing  the 

problem  of  exhaustive  testing  can  be  posed  in  matrix  format.  Since  a 

connection  matrix  C  is  a  descriptor  of  potential  links  between  segments,  the 

execution  sequence  in  response  to  an  input  data  value  x^  (in  most  applications 

x^  is  a  vector  instead  of  a  scalar),  can  be  associated*  with  a  submatrix  of 

C,  say  .  Since  C  is  finite,  the  problem  of  exhaustive  testing  can  be 

framed  as  follows:  for  C  a  given  connection  matrix  find  a  set  x,  ,x„,...x  , 

12m 

so  that  for  the  associated  submatrices  S, ,S0,...S 

12  m 

m 

C  =  B  S, 
i=l  1 


*As  discussed  in  Reference  1,  an  execution  sequence  can  be  mapped  to 
submatrix  by  ignoring  the  ordering  of  its  branches.  This  is  valid  only 
because  of  the  definition  used  here  fo-  exhaustive  testing. 
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where 


m 


represents  the  Boolean  union  or  sum  of  the  .  (This  essentially  defines 
the  nature  of  exhaustive  testing  at  the  track  level). 

An  efficient  test  would  be  one  in  which  the  number  of  test  points, 
m,  is  minimal .  ’ > 

As  noted  above,  essentials  of  the  process  involve  associating  with  each 
decision  point  (two-way)  or  predicate  within  the  program,  a  function  which 
has  a  non-negative  value  when  the  predicate  is  true,  and  negative  value  when 
it  is  false.  In  many  cases,  such  as  comparison  between  program  variables 
by  inequalities,  the  expression  in  the  predicate  can  serve  directly  to  define 
the  function.  If,  for  example,  there  is  a  test  P<Q,  then  the  variable 
assignment,  or  function,  OP-Q,  can  be  used.  Since  the  functions  are 
relations  among  variables,  they  can  be  considered  to  be  program  variables. 

By  forming  variables  of  this  kind  at  each  branch  point,  the  program  is 
augmented  in  such  a  way  that,  in  response  to  an  input  data  set,  an  execution 
sequence  will  take  place  in  which  values  are  given  not  only  to  all  program 
variables  but  also  to  the  augmenting  variables,  which,  as  noted,  are  program 
variables. 

Because  the  signs  (+  or  -)  of  the  augmenting  variables,  set  up  a  unique 
pattern  for  any  input  data,  they  can  be  used  to  define  the  equivalence 
classes  mentioned  above.  It  is  noted  again  that  in  the  formation  of  the 
equivalence  classes  the  ordering  of  the  sequence  has  been  ignored. 

In  a  different  mode  of  usage,  the  sign  of  each  of  the  augmenting 
variables  can  be  specified  in  advance,  and  a  point  (or  region)  in  the  input 
data  space  causing  this  pre-specif ied  pattern  of  signs  can  be  sought.  By 
assignment  of  any  of  a  number  of  simple  objective  functions  of  the  augmenting 

variables,  with  properties  described  subsequently,  the  problem  can  be  stated 
as  a  search  problem  generally  identified  with  optimization  problems. 
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Generally  the  search  is  made  to  find  data  which  will  make  the  objective 
function  positive;  it  is  not  necessary  to  achieve  a  maximum  for  the  objective 
function,  only  that  the  value  of  the  function  be  positive.  This  problem  is 
much  simpler  than  the  optimization  problem. 

The  technique  due  to  W.  Miller  and  D.  Spooner  (Reference  3)  is 
illustrated  by  their  example  shown  below.  Their  description  of  the  example 
has  been  augmented  in  several  ways. 

The  problem  of  the  example  is  one  of  triangularization  of  an  NxN  matrix 
by  Gaussian  elimination. 


The  original  code  is  shown  in  Figure  2.  A  combined  flowchart  and  code 
with  predicates  and  branches  identified,  is  shown  in  Figure  3.  The  predi¬ 
cates,  shown  enclosed  in  rectangular  boxes  are  attached  to  the  node  represent¬ 
ing  the  site  of  their  occurrence.  The  augmented  code  employing  the  functions 
associated  with  the  predicates,  is  shown  in  Figure  4.  The  input  data  to  the 
program  consists  of  the  nine  matrix  entries:  A(1 ,1 ) ,A(2,1 ) . . . ,A(3,3) . 

IP(N)  -  t 

D06K-1.N 

IFIK.EO.NIGO  TO  5 
KP1  -  K+1 
M  =  K 

DO  1  I  -  KP1.N 

IF  <ABS(A(I,K».GT.ABS(A|M,K)))  M  =  I 

1  CONTINUE 
IP(K)  -  M 

IF(M.NE.K)  IP(N)  «  -IP(N) 

T- A(M,KI 
A(M.K)  -  A(K.K) 

A(K,K)  -T 
IF  IT.EQ.O.I  GO  TO  5 
002  1  =  KP1.N 

2  A(I,K)  -  -Ad.Kirr 
DO  4  J  -  KP1.N 

T  ■=  A(M,J) 

A(M,J)  >  AIK.JJ 
AIK.JI  -  T 

IFIT.EQ.O.)  GO  TO  4 
DO  3  I  -  KP1.N 

3  A(I.J)  -  A(I,J)  +  All.KI-T 

4  CONTINUE 

5  IF  (A(K.K).EQ  O.)  IPINt  -  0 

6  CONTINUE 
RETURN 
END 


Figure  2.  Coding  for  Example  Prograrr 
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Formation  of  the  augmented  code  is  accomplished  by  making  a  straight  line 
pass  through  the  program  under  the  assumption  that  the  predicates  inside «the 
DO  loop  1  are  all  true,  and  all  of  the  rest  are  valued  false.  It  is  noted 
that  the  test  results  denoted  as  K=N,  are  governed  by  the  input  assignment 
to  the  matrix  order,  N,  here  taken  in  the  example  as  N=3.  There  is  a  "false" 
valuation  until  K=3.  These  valuations  are  implicitly  made  in  the  construction 
of  the  program  into  a  straight  line  representation.  Similarily  the  tests, 
denoted  as  M=K,  are  completely  determined  by  the  tests  in  the  DO  loop  1  and 
do  not  explicitly  show  in  the  augmented  code;  they  are  used  to  develop  the 
straight  line  code. 

The  variable  ,  shown  on  the  first  line  of  Figure  4,  is  positive  if  the 
predicate,  ABS  (A(2,l )) -ABS  (A(l,l)),  is  true;  and  this  condition  has  been 
specified  as  holding,  since  the  predicate  is  in  the  DO  loop  1.  A  similar 
remark  applies  to  C2  and  Cy  in  the  straight  line  listing  because  they  are 
repeats  of  the  same  test  encountered  under  new  conditions.  On  the  other 


C 


3  " 


c 


4 


c 


5 


c 


8 


c9 

c10  ' 
C11  " 


ABSIAI2.1))  -  ABS(A(1,1)>  >  0 

ABS(A<3.1I>  -  ABSIAI2.1II  >  0 
T-  AI3.1) 

A(3.1)  »  A(1,1) 

All  ,1 )  "  T 

ABS(T)  >  0 
AI2.1)  -  -AI2.1I/T 
AI3.1I  -  -AI3.1I/T 
T-  A  (3.2) 

AI3.2I  -  AI1.2I 
A(1, 2)  -  T 

ABSITI  >0 

AI2.2I  -  AI2.2I  +  AI2.1  }*T 
AI3.2I  -  AI3.2I  +  A(3,1)*T 
T  -  A (3,31 
AI3.3I  -  AI1.3) 

AI1,3)-T 

ABSITI  >  0 

AI2.3I  =  AI2.3I  +  A(2,1  l*T 
AI3,3)-A(3.3)  +  AI3,1)*T 

ABSIAI1.1H  >  0 

ABSIAI3.2I)  ABS(A(2,2I)  >  0 
T  -  A  (3.21 
AI3.2)  =  AI2.2I 
AI2.2I  -  T 

ABS(TI  >  0 
A(3,2)  *  -AI3.2I/T 
T  -  AI3.3I 
AI3.3I  »  A  12,31 
A  ( 2 ,3 1  -  T 

ABSITI  >  0 

AI3.3)  *  AI3.3I  +  AI3,3I*T 
ABSI AI2.2II  >  0 
ABSIAI3.3II  >  0 


Figure  4.  Augmented  Code  for  Example  Program 
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hand,  the  two  tests  shown  in  Figure  3,  denoted  as  T=0,  are  taken  to  be 
false  on  each  encounter,  and  the  value  C^,  C^,  Cg,  Cg  and  Cg  will  all  test 
positive  if  the  false  branches  are  to  be  taken.  (Since  it  is  only  required 
that  T  be  non-zero,  the  C's  could  also  be  chosen  to  be  negative,  but  the 
analysis  is  tailored  around  positive  valuations.) 

It  is,  of  course,  possible  to  express  the  C's  to  explicitly  relate  them 
to  the  input  data.  This  was  done  by  Miller  and  Spooner  threading  back  from 
the  predicate,  where  variable  is  defined,  through  intermediate  assignments 
to  the  original  input  data.  This  is  fairly  simple  because  the  program  is 
straight-lined.  The  technique  is  illustrated  by  a  detailed  analysis  of  the 
auxiliary  variable,  Cy.  In  terms  of  program  variables 

C?  =  ABS  (A(3,2) )-AB$(A(2,2) ) , 

and  these  can  be  traced  through  the  calculations  and  assignments  as  follows: 

substituting  for  A(3,2)  and  A ( 2 , 2 ) , 

C?  =  ABS  (A(3,2)+A(3,l )*T)-ABS(A(2,2)°+A(2,1 )*T) ; 

then,  since  only  A(2,2)°  is  input  data  (and  is  marked  by  a  superscript,  0) 
further  backing  is  required;  since  T  =  A(3,2)^  at  this  point  in  the  program, 
and  A(3,2)^  is  input  data,  the  expression  can  be  written 

C7  =  ABS  (A(3,2)+A(3,1)*A(3,2)°-ARS(A(2,2)0+A(2,1)*A(3,2)°); 

but  A(3,2)=A(1 ,2)^,  A( 3 ,1 ) =-A (3,1)  /A ( 3,1)*^  and  A(3,l)  in  the  numerator  is 
equal  to  A ( 1 ,1 ) 0 .  These  and  similar  substitutions  provide 

C  =  ABS{ A ( 1 ,2)°  -  (A(l ,1 )°/A(3 ,1 ) 0 ) *A ( 3 , 2 ) °] } - ABS{ A ( 2 , 2 ) 0 

-[(A(2,1)°/A(3,1)°)*A(3,2)0]}. 

This  is  an  explicit  representation  of  Cy  in  terms  of  inout. 

This  illustrates  the  difficulties  in  attempting  to  relate  logical  paths  to 
input  da 
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Although  this  process  is  feasible  for  simple  programs,  and  in  many  respects 
resembles  symbolic  execution  in'reverse,  it  presents  the  same  difficulties 
accompanying  the  development  of  equivalence  classes.  An  alternative  is  to  work 
forwardly  from  the  input  data  to  valuations  of  the  C's,  and  their  associated 
predicates.  In  this  procedure,  for  properly  picked  input,  all  of  the  C's  will 
be  positive  and  the  execution  path  will  proceed  along  the  prespecified  path. 

The  new  problem  is  then  one  of  searching  for  areas  rather  than  solving  for 
points.  These  may  seem  to  be  problems  of  the  same  order  of  difficulty  but  they 
are  not.  In  general  applications  the  searching  process  need  not  proceed  to  the 
same  level  of  definition  that  the  solving  process  does.  An  analogy  can  be  made 
with  polynomial  evaluation:  it  is  far  easier  to  locate  a  point  where  a 
polynomial  is  positive,  than  it  is  to  find  a  root  for  the  polynomial . 

3.1.3  Test  Techniques  for  Segment  Coverage 

To  illustrate  some  of  the  characteristics  of  the  test  techniques  employed 
the  problem  discussed  above  is  taken  in  the  framework  of  the  flow  diagram  of 
Figure  5.  The  node  numbers  shown  are  in  a  1-1  relation  to  the  instructions  and 
labels  of  Figure  3.  DO-loops  are  easy  to  identify  by  the  letters  E  (end)  and  S 
(stay),  emanating  from  the  end  of  the  loop.  The  predicates  are  also  easy  to 
identify  by  means  of  the  T  and  F  letters  labelling  the  exits.  The  D01  loop,  for 
example,  starts  at  node  6  and  ends  at  node  9,  similarly  the  D06  loop  starts  at 
Z  and  ends  at  31.  The  specified  path  for  the  sample  problem  can  be  identified 
in  Figure  6.  All  predicate  valuations  (that  are  input  dependent)  are  false 
except  the  one  inside  of  the  D01  loop.  Both  True  and  False  branches  were 
shown  to  be  taken  of  the  nodes  3  and  11,  corresponding  to  the  predicates 
IK  =  N  I  and  IM  =  Kl  .  These  are  not  assigned  auxiliary  variables  but  are  used 
to  straightline  the  program;  as  a  result  they  are  permitted  either  predicate 
valuation. 

Miller  and  Spooner  employ  several  "objective"  functions,  generically  denoted 
f(C-|,C2,...C  );  each  has  the  property  that  f>0,  when  one  or  more  of  the  C's  is 
negative,  and  f>0  when  all  of  the  C’s  are  positive.  As  an  example,  the  function 

F ( C i  ,C£» . . . , Cm ) -mi n(C.|  ,C^,,,  ,  C^ ) 
would  serve  for  that  purpose. 
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The  problem  at  hand,  then,  becomes  one  of  searching  over  the  input  data 
space  for  values  where  f  is  positive  for  the  specified  execution  track.  In 
the  example  problem,  Miller  and  Sponner  start  the  search  with  a  "randomly" 
chosen  matrix 


A 


o 


'3  1 

1  4 

1  1 


1 

1 

5 


which  produces  f  =  -2 

Using  direct  search  methods,  they  derive  a  data  set 


"0.3857 

18.62 

1.0 

0.6268 

-13.865 

1.0 

1.439 

1.0 

5.0 

which  makes  f =0. 241 1 .  According  to  Miller  and  Spooner,  this  is  accomplished 
in  less  than  1  second  of  CPU  time  on  an  IBM  370/168.  The  resulting  coverage 
of  the  program  is  indicated  in  Figure  6.  Because  of  multiple  passes  over 
some  portions  of  the  program,  depiction  is  less  than  perfect.  The  specified 
path,  however,  is  achieved  by  the  data. 

The  usefulness  of  this  procedure  is  best  appreciated  when  used  in 
conjunction  with  a  combination  of  the  "random"  drivers,  suggested  in  the 
earlier  work,  augmented  by  constructed  cases.  The  latter  cases,  are  designed 
to  "fill-in"  for  data  that  were  taken  so  infrequently  by  random  numbers  that 
they  make  the  former  process  uneconomical. 

For  illustrative  purposes,  the  initial  input  data  is  taken  as  the  "random" 
matrix,  used  by  Miller  and  Sponner  to  start  the  process.  For  this  matrix  as 
input,  the  FALSE  branch  out  of  7  is  taken  at  least  once.  Thus,  the  "random" 
start  exercises  a  path  segment  which  the  "optimum"  data  does  not. 
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The  predicates  IT  =  Oi  are  not  true  unless  the  zero-valued  matrix  elements. 
If  the  3x3  zero  matrix  is  used  for  data,  all  tests  T=0,  as  well  as  the  final 
A(K,K)=0  are  true,  and  the  constructed  case  produces  the  execution  track 
shown  in  Figure  7. 

Additional  tests  for  programs  can  often  be  suggested  by  some  built-in 
symmetries  in  the  problem.  Thus,  for  a  short  problem  it  can  be  generally 
assured  that  when  input  data  is  permuted,  the  resulting  execution  tracks  will 
be  different.  When  a  polynomial  solver  is  employed  it  is  well  known  that  a 
set  of  symmetric  relations,  involving  the  roots,  define  the  coefficients  of  the 
polynomial.  Further,  there  are  relations  between  the  coefficients  of  a 
polynomial  and  the  polynomial  whose  roots  are  shifted,  squared,  and  inverted. 

In  the  present  instance  of  a  matrix  triangularization,  the  interchange  of  two 
rows  can  be  expected  to  cause  a  different  response. 

As  a  matter  of  interest,  when  the  matrix  obtained  by  the  optimization 
process  is  used  with  its  1st  and  2nd  rows  interchanged,  the  resulting  track  is 
shown  in  Figure  8.  The  False  branch  out  of  node  8  is  taken  on  the  first  entry, 
and  the  True  branch  on  (one  or  more)  subsequent  passes.  (In  the  particular 
sequence  of  tests  employed,  there  is  nothing  new  added  by  this  test). 

For  the  simple  problem  illustrated  here,  all  segments  of  the  program 
are  tested  by  the  three  cases  consisting  of  the  starting  "random"  matrix,  the 
zero  matrix,  and  the  matrix  obtained  by  the  optimization  procedure. 

The  method  suggested  in  the  illustration  leads  to  extensions  of  value 
to  the  general  problem  of  exhaustive  testing.  As  noted  in  earlier  work,  the 
problem  of  testing  a  program,  only  to  the  point  where  every  instruction  and 
every  branch  has  been  executed,  is  generally  a  computationally  small  enough 
problem  making  it  feasible  for  almost  any  program.  This  is  true  basically 
because,  for  a  minimum  with  k  predicates  (two-way),  there  are  no  more  than 
2k  data  points  required  to  "test"  the  program  in  this  way,  whereas  there  are 
as  many  as  2  differential  logical  paths  (many  more  if  loops  are  permitted). 
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Figur«  8.  Raspons#  to  Data  Formed  by  Interchanging  1*t  and  2nd  Rows  of  Original  Matrices 


For  the  general  testing  problem,  a  sequence  of  random  numbers  or  vectors 
many  be  used  to  develop  a  set  of  tracing  vectors  whose  components  represent  the 
Boolean  valuations  of  the  C's.  These  runs  would  generally  be  both  inexpensive 
and,  because  they  are  the  first  to  be  employed,  would  be  of  high  yield.  After 
a  reasonably  large  set  of  random  numbers  have  been  run,  the  sec  of  associated 
vectors  (as  distinct  from  the  values  of  the  auxiliary  function  exemplified  by 
f  in  the  above  discussion)  can  be  examined. 

Except  for  cases  where  predicates  involve  equality  between  expressions 
involving  program  variables,  the  vectors  can  be  collected  on  the  basis  of  com¬ 
ponent  comparisons.  Thus,  if  there  are  both  zeros  and  ones*  in  the  first 
component  position,  the  testing  has  "exhausted"  the  cases  provided  by  the  first 
predicate. 

A  simple  sorting  procedure  will  identify  unexercised  branches.  In  case 
specific  predicates  are  not  represented  by  both  "true"  and  "false"  values, 
the  process  described  above  can  be  used  to  search  for  data  that  will  force 
the  program  in  the  desired  way.  Should  there  be  neither  valuation,  the  same 
general  procedure  can  be  used  initially. 

It  is  possible  in  this  scenario,  that  the  so-called  "scaling  problem",  a 
result  of  non-common  scales  on  the  variables  involved  which  tends  to  confuse 
some  optimization  problems,  can  be  used  to  advantage  in  the  case  of  a  search 
for  data  to  exercise  a  specific  branch.  For  example,  if  a  variable  associated 
with  a  predicate  is  more  sensitive  by  multiplication  or  division  of  appropriate 
factors  employed  in  its  definition,  then  strong  responses  will  occur  with  only 
small  changes  in  input.  A  sequence  of  applications  of  such  factors  to  each 
localized  variable  would,  probably,  produce  good  coverage. 

3.1.4  Test  Techniques  for^  Tra cM  cvoi  coverage 

The  major  use  of  the  above  technique  is  in  establishing  exhaustive  tests 
for  a  given  program  package.  The  utility  as  a  software  metric  is  clear.  As 


*A  blank  would  indicate  no  test. 
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noted  in  Reference  1,  one  quality  of  Software  having  universal  appeal,  is  the 
degree  to  which  a  problem  has  been  tested.  Ideally  this  would  be  measured  in 
terms  of  the  ratio  of  the  number  of  logical  paths  executed  by  all  tests  per¬ 
formed  on  the  package,  to  the  total  number  of  paths  present.  However,  the 
latter  is  almost  never  known,  and  there  are  many  non-real izable  paths  which  are 
not  apparent;  even  the  realizable  ones  may  not  be  easy  to  enumerate.  Thus  the 
more  easy  to  obtain  ratio  is  a  substitute. 

Reference  1  describes  the  method  of  estimating  the  total  number  of  tracks 
realizable  by  random  numbers.  This  method  depended  on  the  development  of  the 
count  of  the  number  of  trials  between  discovery  of  new  paths.  An  asymptotic 
limit  to  the  total  was  then  developed  on  the  basis  of  an  algorithm.  This 
technique  could  be  applied  to  individual  branches  or  to  any  selected  set  of 
branches.  Some  measure  of  the  degree  to  which  a  program  has  been  tested 
may  be  developed  from  the  combination  of  the  yields  obtained  by  using 
constructed  cases  and  from  application  of  random  numbers.  In  specific 
production-type  applications,  studies  of  so-called  impossible  pairs  may  be 
made  but  for  development  of  a  universal  metric,  such  a  fine-grained  investi¬ 
gation  is  not  warranted. 

In  order  to  automate  the  track-level  testing  procedure  several 
modifications  to  the  APTS  were  made  and  a  post  processor  of  data  was 
programmed. 

First,  the  algorithm  which  obtains  the  estimated  number  of  tracks 
through  a  program  obtained  by  using  random  numbers  as  program  drivers  was 
programmed  as  part  of  a  post  processing  routine.  This  problem  had  been  solved 
in  principle,  but  implementation  of  it  heretofore  had  been  effected  by  the 
tedious  process  of  desk  checking  segment  usages  against  all  past  usages. 

The  selection  of  random  values  for  the  input  variables  (real  or  integer) 
provides  the  set  of  values  for  one  run.  The  procedure  employed  for  estimating 
the  number  of  tracks  that  will  be  exercised  requires  a  number  of  executions 
and  comparisons.  In  the  automatic  version,  the  track  that  accompanies  one 
input  data  (random)  selection  is  identified  in  terms  of  a  zero  or  one  assign¬ 
ment  to  the  arbitrarily  ordered  set  of  segments  which  comprise  the  list  of 
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segments:  a  zero  for  nonusage  and  a  1  for  one  or  more  usages.  (Two  paths 
which  differ  in  their  nonzero  counts  of  the  usages  of  segments,  or  in  their 
order  of  execution,  are  considered  to  have  the  same  track). 


In  the  implementation  of  the  estimation  process,  the  above  outlined  initial 
portion  is  followed  (in  the  postprocessor)  by  a  routine  which  compares  the 
sequence  of  binary  n-tuples  (one  "ordinate"  for  each  program  segment)  in 
order  to  accomplish  two  things: 

A.  Establish  whether  a  newly  examined  track  is  the  same  as  some  track 
earlier  examined,  effected  by  comparing  the  n-tuples  ordinate  by  ordinate 
against  all  previously  taken  tracks, 

B.  Marking  the  trial  number  of  the  current  track  by  a  zero  or  1  in 
accordance  with  the  results  of  the  comparisons,  a  zero  if  an  "old"  n-tuple 
has  been  found  and  a  1  if  the  examined  track  is  new. 

The  data  for  the  estimation  procedure  consist  of  the  pattern  of  0’s  and 
I's  obtained  in  the  above  comparisons.  The  primary  observable  consists  of  the 
total  trials  between  adjacent  l's.  These  spacings  between  I's  are  reported 
as  X-j ,  X2,...,Xn  and  represent  the  difference  in  the  indices  representing 
trial  numbers:  X-j  is  the  separation  between  the  first  trial  number  (by  defi¬ 
nition,  the  first  trial  results  in  the  first  new  track)  and  the  trial  number 
which  produces  the  second  new  track  (usually  tHs  separation  is  1  because  of 
the  high  likelihood  that  a  new  data  set  will  produce  a  different  track);  X^  is 
the  separation  between  the  third  and  second  new  track,  etc. 

With  data  X^ ,  X2,...,Xn  obtained  by  running  the  program  over  T  trials, 
the  number  of  new  tracks  is  estimated  from  the  equation 


n 


L 

i=l 


_ 1 _  =  _ nT 

N-(i-I }  ”  n 

NT  -  £  (i-l)X. 
i  =  l  1 


where  N  is  the  unknown,  X.  are  as  defined,  T  is  the  total  number  of  trials  and 
n  is  the  number  of  X.  employed. 


The  augmented  version  of  PTS  achieves  this  entire  process  of  comparison  and 
estimation  automatically. 
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3.2  APPLICATIONS 


3.2.1  Air  Force  Logistics  Mode1--0RLA 

In  order  to  avoid  the  algorithmic-type  programs  previously  studied, 
programs  which  are  more  typical  of  those  encountered  in  the  field  were 
reviewed,  specifically  the  Air  Force  Logistics  programs  were  reviewed.  Inspec¬ 
tions  of  several  programs  were  made  for  the  purpose  of  selecting  a  useful 
candidate  for  coverage  testing.  A  review  of  the  MOD-METRIC  model  revealed 
a  very  complex  program  which  would  have  provided  an  excellent  candidate  because 
of  the  diverse  modes  which  can  be  exercised.  However  the  fact  that  documen¬ 
tation  of  the  FORTRAN  program  is  almost  non-existent  in  the  mid-levels  of 
documentation  (between  the  overview,  on  the  one  end,  and  inserted  comments, 
on  the  other),  the  program  was  passed  over.  The  LSC  (Logistics  Support  Cost) 
model  was  not  selected  because  it  consists  of  a  set  of  rather  simple  algebraic 
formulas.  Another  model,  LEM  (Logistics  Effect  Model)  is  not  yet  widely  known 
in  the  Air  Force,  and  primarily  was  eliminated  for  that  reason.  The  Air  Force 
LCOM  (Logistics  Composite  Model)  was  investigated  and  while  its  basic  or 
underlying  language  is  FORTRAN,  it  has  a  language  of  its  own  and  is  not  there¬ 
fore  suitable  for  analysis.  Another  difficulty  with  LCOM  is  that  production 
runs  with  that  model  would  cost  far  in  excess  of  any  contemplated  expenditures 
for  the  testing  task  which  was  planned.  This  is  so  because  the  model  relies 
on  simulations  with  an  underlying  SIMSCRIPT  II  program,  to  produce  Monte  Carlo 
based  statistics  of  operational  parameters.  The  program  with  greatest  poten¬ 
tial  among  those  investigated  is  commonly  called  ORLA  (Optimum  Repair  Level 
Analysis).  The  particular  version  employed  was  written  by  0.  R.  Johnson  of 
Warner-Robins  Air  Force  Logistics  Center. 

ORLA  employs  costs  associated  with  the  acquisition,  logistic  support, 
and  replacement,  or  airplane  subsystems.  Three  options  are  generally  con¬ 
sidered  in  an  ORLA  analysis:  discard  at  (suspected)  failure  of  the  subsystem; 
repair  of  the  failed  subsystem  at  the  base  (home  airport),  or  repair  at  an 
Air  Force  depot  (generally  supporting  several  bases).  Some  11  different  cost 
components  are  involved  for  the  latter  two  options,  while  3  cost  components 
comprise  the  discard  option  total.  Although  computations  are  not  complex, 
and,  indeed,  the  cost  components  are  simply  algegraic  formulas,  the  so-called 
sensitivity  analysis  presents  some  interesting  complexities  and  decisions. 
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The  aim  of  this  sensitivity  analysis  is  to  determine  (to  the  nearest  1%  of 
the  baseline  value)  the  point  at  which  the  nominal  decision,  derived  from 
the  baseline  values,  will  be  reversed.  This  is  accomplished  for  any  choice 
from  the  17  different  input  factors,  and  it  provides,  as  the  name  indicates, 
a  measure  of  the  sensitivity  or  stability  of  the  decision  in  the  face  of 
possible  changes  in  or  misestimation.  The  sensitivity  analysis  is  flow¬ 
charted  in  Section  3. 2. 1.1  where  the  application  to  the  ORLA  program  is 
illustrated. 

3. 2. 1.1  Segment  Level  Coverage  of  ORLA 

The  main  ORLA  program  consists  of  488  lines  of  FORTRAN  code  (each  branch 
of  branching  instructions  are  counted).  Briefly  the  components  of  ORLA  can 
be  described  by  the  following:  Initialization  (about  15  instructions);  Read 
Constants  (64);  Compute  Failure  Rate  (59);  Computation  of  Aerospace  Ground 
Equipment  Usage  (66);  ORLA  Variable  Identification  (34);  Economic  Analysis 
(33);  Write  Summary  (15);  Computation  Routine  (58);  Rank  Economic  Values  (22); 
Sensitivity  Analysis  (93);  Write  Repair  Sunmary  (12).  (In  addition  three 
peripheral  and  non-essential  subroutines  are  included  in  the  program:  two 
are  merely  messages  for  the  user  in  case  he  requires  explanations  of  the 
program,  the  third  is  set  of  error  messages  in  case  of  inconsistencies  in 
the  data.  These  subroutines  are  not  included  in  the  discussion  which  follows.) 
A  listing  of  an  APTS-augmented  ORLA  is  given  in  Appendix  A. 

To  drive  the  basic  ORLA  a  total  of  54  variables  are  employed.  These 
variables  provide  descriptions  of  all  the  logistics  involved  in  acquiring, 
shipping,  repairing,  maintaining,  and  resupplying  an  aircraft  subsystem. 
Included  are  variables  which  represent  overhead,  such  as,  training  of  main¬ 
tenance  personnel,  management  of  ir.entory,  and  facilities.  The  54  variables 
are  divided  into  2  main  classes.  First  a  set  of  36  variables  describe  the 
rates  which  hold  or  are  projected  to  hold  for  the  time  of  the  analysis,  the 
force  size  and  deployment  scheme,  labor  and  material  rates,  and  so  forth. 

In  addition  to  these,  a  second  class  bears  directly  on  the  item  or  subsystem 
analyzed  (ORLA'd);  there  are  17  variables  in  the  class  and  they  describe, 
cost  and  weight  of  the  subsystem  and  its  parts,  repair  time,  and  the  docu¬ 
mentation,  training,  and  special  facilities  which  are  required  for  the  item. 
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In  addition  to  these  basic  variables  there  are  10  additional  variables 
which  are  derived  from  intermediate  computations  which  rely  either  on  keyboard 
entry  (of  parameters  relating  to  the  MTBF)  or  on  sharing  of  resources  by 
several  items  (AGE  or  test  equipment  which  is  employed  or  several  different 
subsystems  of  the  aircraft  for  example).  The  reason  for  identifying  them 
with  the  input  variables  is  that  they  also  can  be  subjected  to  the  sensitivity 
analysis. 

As  noted  earlier  the  ORLA  program  employs  the  input  values  associated 
with  a  given  item  and  computes  the  costs  which  would  be  incurred  under  the 
three  options  (discard,  repair  at  base,  repair  at  depot).  On  the  basis  of 
the  three  ranked  costs  the  optimum  or  least  cost  repair  level  can  be  determined. 
Although  the  numerical  values  of  the  costs  of  the  various  components  of  cost 
are  printed  out  and  an  indication  of  the  assurance  or  firmness  of  the  decision 
which  the  program  makes  can  be  made  from  these  magnitudes,  a  better  measure 
of  the  firmness  of  the  decision  can  be  made  by  use  of  sensitivity  analyses. 

Each  run  a  set  of  up  to  10  user-selected  variables  can  be  identified  for  use 
in  this  analysis.  As  noted  before,  the  primary  purpose  is  to  determine,  from 
variations  in  the  costs  due  to  changes  in  the  selected  variable,  the  point 
(a  percentage  of  nominal  value]  where  the  decision  based  on  nominal  or  baseline 
costs  is  reversed.  This  is  determined  to  the  nearest  percentage  on  the  range 
20%  to  500%  (1/5  to  5  times  nominal).  Should  no  change  in  decision  occur 
over  this  range,  the  decision  is  clearly  stable  with  respect  to  the  variable 
inspected. 

Certain  variables  are  known  to  affect  certain  options  more  than  others 
and  a  user  wishing  to  test  for  coverage  could  be  guided  by  this  a  priori 
knowledge.  Some  of  this  kind  of  knowledge  is  also  used  in  the  construction 
of  cases  which  are  discussed  here.  This  is  countei  to  the  mode  which  would 
be  used  in  the  final  testinq  scheme  where  it  is  assumed  the  user  is  unaware 
of  the  relationship  between  input  and  any  particular  program  segment.  In  the 
final  version  each  variable  would  be  varied  at  random  to  provide  an  initial 
coverage;  subsequent  coverage  would  be  initiated  by  a  specification  of  a 
program  path  or  track,  then  continued  by  invoking  a  search  procedure  on  the 
input  data,  and  hopefully  consumated  by  an  identification  of  a  point  which 
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produces  an  execution  which  includes  selected  path  or  track.  Because  the 
status  of  the  study  has  not  progressed  to  the  point  where  automatic  insertion 
and  data  generation  are  possible  the  procedure  used  in  the  example  relies 
on  knowledge  of  the  program. 

It  is  cumbersome  to  illustrate  the  usage  of  APTS  on  the  entire  ORLA  pro¬ 
gram,  but  a  good  indication  of  the  way  APTS  can  be  applied  in  static  analysis 
can  be  provided  by  use  and  inspection  of  a  compact  portion  of  the  listing. 

In  Figure  9  is  a  flow  chart  of  the  portion  of  the  program  called  Reversal 
Analysis.  This  is  used  in  part  of  the  sensitivity  analysis  to  compare  and 
rank  the  costs  of  the  three  options.  For  convenience  the  ORLA  program  with 
segments  identified  comprises  Appendix  A. 


Application  of  APTS  in  static  composition  of  segments  from  the  coding  of 
the  above  identified  program  portion  is  effected  by  first  numbering  the  instruc¬ 
tions  as  shown  in  Figure  10.  This  shows  the  numbering  in  the  leftmost  column 
and  these  are  associated  with  the  instruction  on  the  right.  Labels  shown 
correspond  to  the  original  listing  and  are  employed  in  the  flowchart  of 
Figure  9.  Thus  396  corresponds  to  the  labelled  (215)  instruction,  JSEN ( 1 )=KDT, 


at  the  top  of  Figure  9,  415  corresponds  to  the  predicate,  |  NUMK(1 )-NUMJ(1 )=0 


which  appears  just  after  the  labelled  310  CONTINUE  instruction  in  Figure  10. 
The  APTS  segmentation  of  the  program  in  the  above  described  region  is  shown 
in  Figure  1 1 . 


It  is  important  to  note  that  in  most  cases  the  segments  are  made  up  of 
several  of  the  PTS  segments  defined  in  Reference  1.  Those  segments  were 
truncated  by  labels,  GOTO's,  etc.  Several  other  points  require  explanation. 
First  the  segments  identified  with  the  bracket/parenthesis,  start  with  an 
instruction  number  which  is  either  the  start  of  the  program  or  subroutine, 
or  a  predicate  (IF  statement  in  most  cases),  the  remainder  of  the  numbers 
in  the  sequence  denote  the  instructions  which  will  be  executed  in  sequence, 
the  end  of  the  sequence  of  numbers  is  identified  by  a  number  corresponding 
to  a  predicate  or  branch  point.  Thus  T^  starts  with  the  labelled  instruction 
396,  then  in  turn  by  397,  398,  399,  400  and  ends  with  401 
401  is  an  implied  predicate. 


The  instruction 
If  the  predicate  is  true 
the  next  segment  taken  is  T^  which  describes  passage  from  the  D0210  loop  to 


00  LOOP  END=TRUE 


MC0O««(U  DOI/OL4 


59 


306  HOLD  -  JSEN  (IB) 

JSEN  (IB)  -  JSEN  IIZ) 

JSEN  (IZ)  -  HOLD 
HOLD  •  NUMJ  (IB) 

NUMJ  (IB)  *  NUMJ  IIZ) 
NUMJ  (IZ)  -  HOLD 


- - I  NUMK  (1)  -  NUMJ  111  •  0 


222  PCT  -  PCT  ♦  0.90 


VAL  (1C)  •  ORIG 


GO  TO  'SENSITIVITY  ANALYSIS' 


<  716  'CHECK  FOR  REVERSAL' 


Figure  9.  Revarial  Analyiii 
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REVERSAL  ANALYSIS 


396 

215  JSEN(l)  =  KDT 

397 

JSEN(2)  -  KFT 

398 

JSEN(3)  -  KU¬ 

399 

DO  210  I  *  1,3 

400-401 

210  NUMO(I)  *  I 

402 

300  00  310  IB  *  1,2 

403 

K  “  IB  +1 

404 

DO  310  IZ  *  K.3 

405 

305  IF  (JSEN(IB)-JSEN(IZ)) 

406 

306  H0LD=JSEN( IB) 

407 

OSEN(IB)  -  OSEN(IZ) 

408 

JSEN(IZ)  =  HOLD 

409 

HOLD  «  NUMJ(IB) 

410 

NUMJ( IB)=NUMJ(  IZ) 

411 

NUMJ( IZ)=H0LD 

412-414 

310  CONTINUE 

415 

IF  (NUPSK(l)  =  NUMJ(l)) 

416 

228  CONTINUE 

417 

IF  (IP-8)  322,222,229 

418 

222  PCT  *  PCT-.90 

419 

GO  TO  2300 

420 

322  PCT  =  PCT-.l 

421 

GO  TO  2300 

422 

229  PCT  =  PCT  +  .1 

423 

GO  TO  2300 

424 

320  CONTINUE 

425 

IF  (IP-8)  375,375,  360 

426 

375  PCT  =PCT  +  .09 

427 

GO  TO  340 

428 

360  PCT  =  PCT  -.09 

429 

340  OX  *  ORIG  *  PCT 

430 

VAL(IC)  =  OX 

431 

QCTGM(IPASS)=VAL(31)  * 

432 

ASSIGN  715  TO  JUMP 

433 

GO  TO  100 

310,310,306 


320,228,320 


VAL(32)  *  VAL(46)/VAL(56) 


Figure  10.  APTT  Numbering  for  Progrem 
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MDAC  SEGMENT  XLATOR 


£396-401 ) 

[401,399-401) 

[401-405) 

[405,412-413) 

[413,404-405) 

[413-414) 

[414,402-405) 

[414-415) 

[415,424-425) 

[425-427,429-433,304-319) 

[425,428-433,304-319) 

[415-417) 

[417,420-421,460-461) 

[417-419,460-461) 

[417,4220-429,460-461) 

[405-413) 


Figure  11.  APTS  Segment  Identification 
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the  00310  loop,  and  continuation  to  the  next  predicate  which  is  an  explicit 
predicate,  JSEN(IB-JSEN(IZ)=0  .  If  the  predicate  is  false,  the  segment  T2 
is  executed,  with  an  initial  401,  the  entry  or  reentry  into  the  00210  loop 
at  399. 

Because  of  the  selection  of  only  a  portion  of  the  program,  some  segments 
shown  in  Figure  11,  such  as  T-jq,  list  instructions  which  are  outside  the  range 
of  those  shown  in  Figure  10.  The  explanation  for  T^q  which  will  be  given  here 
should  serve  for  others  as  well.  is  made  up  of  (425-427,  429-433,  304-319), 
and  it  is  the  last  group  that  is  out  of  range.  Instruction  433  is  a  GOTO  100 
instruction  and  the  APTS  number  304  corresponds  to  the  label  100,  which  is  the 
start  of  the  so-called  Computation  Routine.  This  routine  computes  cost  compo¬ 
nents  for  the  three  options  and  the  304-319  segments  is  the  initial  segment 
of  that  routine.  A  "return"  to  the  portion  which  is  displayed  in  Figure  9  is 
at  the  end  of  the  Computation  Routine  (at  APTS  number  355-not  shown).  There 
is  one  entry  point  from  the  Computation  Routine  and  that  is  at  T-j .  So  far  as 
the  local  analysis  is  concerned  T^  joins  to  T^.  The  original  set  of  segments 
can  be  tailored  so  as  to  exhibit  only  local  connections  by  the  above  method. 

So  far  as  the  illustration  of  technique  is  concerned,  however,  there  is  no 
need  to  work  at  that  level  of  detail. 

Figure  9  shows  the  two  major  exits:  computation  routine  (label  100  and 
APTT  number  304);  sensitivity  analysis  (label  2360,  APTS  Number  384).  The 
program  was  initially  driven  with  a  set  of  standard  elements  for  one  of  the 
Air  Force's  aircraft  types  and  an  imaginary  subsystem.  The  program’s  pre¬ 
selected  variables  were  used  in  the  sensitivity  analysis  (these  correspond 
to  the  size  of  the  force  being  fitted,  the  number  of  hours  per  month  the 
aircraft  will  be  used,  the  repair  manhours,  the  unit  cost,  the  Mean  Time 
Between  Failures,  the  cost  of  depot  AGE,  and  the  cost  of  base  level  AGE). 

The  initial  data  exercised  the  segments  listed  in  Figure  11  as  follows. 

Number  of  Executions 
118 
236 
118 
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Segment 


Number  of  Executions 


T4 

241 

T5 

118 

T6 

236 

T7 

118 

T8 

118 

T9 

5 

T10 

3 

T11 

2 

T12 

113 

T13 

28 

T14 

4 

T1 5 

81 

T16 

113 

For  this  arbitrary  set  of  data  ail  explicit  and  implicit  predicates  were 
exercised.  This  complete  (local)  testing  was  fortuitous  in  a  sense,  for  in 
three  successive  runs  with  other  data  T^0  was  not  exercised,  while  Tg  and 
were  not  exercised  in  one  case. 

The  static  aspects  of  APTS  are  well  illustrated  by  the  foregoing.  The 
dynamic  aspects  can  be  illustrated  by  the  results  from  four  data  sets.  The 
first  or  nominal  is  the  set  identified  above,  the  second  maintained  the  same 
standard  elements  and  changed  one  item  parameter,  the  unit  cost  (from  3600  to 
36).  The  third  restored  the  unit  cost  to  the  original  value  and  changed  one 
standard  element,  depot  labor  rate  (from  12.44  to  1).  The  fourth  changed 
the  turnover  rate  from  0.15  to  15. 

Results  over  the  entire  113  segments  of  the  program  show  that  the  initial 
choice  of  data  was  indeed  exceptional,  since  96.63%  (86  out  of  89)  of  the 
segments  exercised  by  the  four  segments  were  exercised  by  the  initial  set. 
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The  change  in  cost  by  a  factor  of  TOO  (the  second  case)  exercised  two 
segments  not  exercised  by  the  first  set  and  these  correspond  to  predicate 
branches  caused  by  the  re-ranking  of  the  costs  of  the  three  options  (discard 
would  be  the  least  expensive).  Similarly  for  the  fourth  set,  a  reversal  of 
the  costs  of  depot  and  intermediate  repair  is  effected  by  the  extreme  value 
chosen  for  depot  turnover. 

Examination  of  the  113  segments  comprising  the  ORLA  program,  shows  that 
24  segments  were  unexercised  by  the  four  simple  cases.  But,  of  these,  13 
depend  on  choices  which  are  prompted  by  the  program;  that  is  they  are  yes/no 
responses  to  questions  concerning  choices  as  to  whether  the  user  wishes  to 
correct  an  entry,  whether  he  wishes  an  explanation,  whether  he  wants  to  run 
a  batch  of  several  items,  etc.  In  some  cases  these  choices  reflect  into  the 
substance  of  the  program  and  in  others  they  stimulate  isolated  calls  and 
returns  without  exercising  any  computations.  Of  the  11  segments  which  remain, 
all  but  four  can  be  exercised  with  data. 


As  a  very  simple  and  brief  explanation  of  the  actual  technique  used  for 
constructed  cases,  and  as  a  useful  means  of  discussion  of  the  automated  version 
of  the  process,  the  predicate. 


E0Q<A|  ,  involving  the  two  program  variables 


EOQ  and  A  will  be  discussed.*  The  APTT  post-processor  tally  usages  of  the 
entire  program  shows  that  the  true  branch  of  this  predicate  is  taken  on  every 
encounter  (1143  passages  in  the  4  cases).  The  code  contiguous  to  the  predicate 
shows  that  the  true  branch  corresponds  to  the  inequality: 


4.4  /A  <  A 


or 


19.36  <  A 


*These  variables  occur  in  the  Computation  Subroutine  and  represent  Economic 
Order  Quantity  (EOQ)  and  a  "Pipeline"  content  (A). 
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Again  by  use  of  other  parts  of  the  code,  it  is  established  that 


fl-  ,2-V45-V48V31'VVW56 


where  the  V's  are  all  input  variables. 

Thus  the  choice  =  0,  among  many  others,  will  cause  the  false  branch 
to  be  taken. 

It  is  well  to  note  that  in  the  contemplated  scheme,  random  numbers  would 
be  used  over  convenient  ranges  for  all  of  the  input  variables,  and,  in  this 
case,  the  probability  of  producing  an  A  value  less  than  19.36  would  be 

extremely  high.  Thus  it  is  highly  likely  that  the  case  investigated  here 

would  not  have  arisen  in  the  context  of  an  unexercised  branch  at  a  correspond¬ 
ing  stage  of  testing,  and  in  fact,  when  100  cases  were  run  this  branch  was 

indeed  executed. 


Should  a  similar  predicate  branch  be  untested  after  an  initial  set  of 
data  runs,  the  following  procedure  would  apply.  The  augmenting  program 
variable  OEOQ-A  would  be  inserted  at  the  predicate  site  during  the  APTS  pre¬ 
processing.  During  each  pass  the  value  of  C  would  be  evaluated  (in  combination 
with  other  inserted  augmenting  variables  at  other  sites  of  predicates).  Varia¬ 
tions  on  the  input  data  would  be  made  according  to  a  search  scheme  until  a 
point  is  reached  where  all  augmenting  variables  have  the  desired  sign  -  in 
the  present  case,  C  must  be  positive. 

The  more  extensive  test  of  ORLA  comprised  a  run  of  size  100.  Several 
interesting  problems  arose  in  the  process  of  obtaining  these  runs. 

Most  of  these  problems  concerned  character  string  inputs.  To  test  in  a 
random  way,  the  variables  of  ORLA,  the  user  must  become  somewhat  familiar  with 
the  sites  where  meaningful  input  is  done,  and  what  type  of  input  is  expected. 
There  are  five  types  of  input  required  by  ORLA: 

1)  real  variable  containing  either  "yes"  or  "no" 

2)  real  variable  containing  real  values 

r 
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3)  integer  variable  containing  integer  values 


4)  double  precision  variable  containing  an  a8  string 

5)  double  precision  variable  containing  one  of  sixty-four  possible 
a8  string  names 

Because  FORTRAN  allows  character  strings  to  appear  in  all  data  types, 
trying  to  recognize  inputs  and  generate  random  values  for  them  causes  a  major 
problem.  After  the  sites  for  inputs  from  the  user  were  established  they  were 
replaced  by  a  call  to  a  hand-generated  input  routine  of  the  proper  type. 

It  was  decided  to  run  one  hundred  test  cases  using  the  random  inputs  as 
test  values.  The  ORLA  source  program  was  pre-processed  by  APTS  and  compiled, 
then  linked  to  the  random  input  routines.  One  hundred  executions  of  the 
instrumented  program  followed.  For  each  execution,  an  output  report  was 
generated  by  ORLA  and  a  post-processor  report  was  generated  by  APTT.  A  log 
was  also  kept  for  each  test  case  run.  There  were  five  types  of  run-time 
errors  that  were  detected  by  the  FORTRAN  run-time  library. 

1)  Floating  point  divide  check 

2)  Floating  point  overflow 

3)  Square  root  of  a  negative  number 

4)  Integer  overflow 

5)  Illegal  character  in  data 

Each  of  these  errors  is  not  an  expected  output  of  the  ORLA  program.  At 
this  point,  an  interesting  point  should  be  made  about  program  testing.  To 
facilitate  the  testing  of  computer  programs  where  there  is  a  possibility  of 
run-time  errors,  either  fatal  or  non-fatal,  there  must  be  a  mechanism  for 
gathering  the  statistics  that  have  been  collected  up  to  the  point  of  the 
error.  Fortunately  the  DEC-10  operating  system  has  such  a  facility. 
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After  one  hundred  test  case  executions,  only  four  segments  failed  to 
be  executed: 

1)  Segment  86  [457,461),  To  execute  this  segment  there  must  be  a 

premature  end  of  file  on  FORTRAN  logical  unit  IW0RK2.  This  appears 

to  be  impossible  because  the  loop  which  reads  the  data  from  this 
file  is  controlled  by  a  variable  that  is  incremented  for  each 
write  to  IW0RK2.  (See  line  277.) 

2)  Segment  96  [199-204),  To  execute  this  segment  the  variable  ITAGE 

must  be  less  than  two  and  the  AGE  SUMMARY  option  must  have  been 

selected. 

3)  Segment  98  [187,200-204),  This  segment  appears  to  be  impossible  to 
execute  under  all  input  values.  If  variable  ITAGE  is  greater  than 
two  and  the  AGE  SUMMARY  option  has  been  selected  then  the  loop 
from  statements  143  to  199  would  be  exited  at  statement  169  before 
segment  98  has  a  chance  to  be  executed. 

4)  Segment  104  [64-64,62-64),  This  segment  was  not  executed  due  to 
the  restricted  range  of  values  selected  for  random  input  to  varia¬ 
ble  IT.  If  the  range  had  been  expanded  from  (0,10)  to  (0,11)  then 
segment  104  should  have  been  hit. 

Thus  segment  coverage  by  the  1 00  test  cases  was  essentially  complete. 

Two  of  the  segments  are  apparently  impossible  to  execute,  and  two  require  user 
options  which  could  be  taken  but  were  not.  The  comprehensiveness  of  the 
random  number  testing  is  clearly  demonstrated  in  this  example. 

3. 2. 1.2  Track-Level  Coverage  of  0RLA 

The  100  test  cases  which  were  used  for  the  segment  coverage  testing  were 
also  employed  in  the  analysis  of  track  coverage.  This  number  turns  out  to  be 
inadequate  for  this  purpose  but  the  difficulties  which  were  described  in  the 
previous  subsection  proscribed  any  attempt  at  exhaustive  testing.  The  fact 
is  that  98  out  of  the  hundred  tracks  generated  were  unique.  This  relatively 
simply  formula-oriented  program  requires  a  test  sample  of  at  least  100  different 
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runs  and  as  indicated  below  the  probability  is  high  that  several  hundred  or 
several  thousand  may  be  required.  This  is  in  stark  constrast  to  the  fact  that 
almost  all  segments  have  been  covered. 

For  ordinary  programs  there  would  not  be  any  problem  for  generation  of 
random  input  is  "from  the  top"  and  can  be  inexpensively  provided,  whereas, 
for  the  interactive  ORLA,  requests  for  input  must  be  responded  to  by  on-line 
monitoring,  resulting  in  constant  attention  and  manual  input  of  information. 

Nonetheless,  the  procedure  of  track  estimation  can  be  well  illustrated 
by  considering  the  initial  segments  of  the  ORLA,  and  sequentially  increasing 
its  size  from  15  to  75  in  steps  of  15.  Estimates  are  made  on  these  segments 
to  produce  trend  data. 

In  Table  I  (two  parts)  the  segment  usages  of  the  complete  ORLA  program 
are  shown.  This  program  consists  of  113  segments  in  the  main  program.  These 
correspond  to  the  first  38  octal  numbers  to  the  left  of  the  arrow  between  the 
38th  and  39th  number.  Those  to  the  right  of  the  arrow  represent  subroutines. 
Each  of  the  first  37  octal  numbers  represents  usages  of  three  consecutive  seg¬ 
ments.  An  octal  digit  of  5  in  the  first  position  indicates,  for  example, 
usage  by  the  1st  and  3rd  segments  and  non  usage  of  the  2nd,  a  7  indicates 
usage  by  all  three  segments.  This  coding  is  continued,  each  representing 
3  consecutively  listed  segments.  The  38th  digit  represents  a  mix  of  the 
112th  and  113th  segment  of  the  main  program  and  the  first  segment  of  the 
first  subroutine  (which  is  immaterial ) . 

It  is  noteworthy  that  the  subroutines  of  the  program  have  apparently 
or  probably  been  fully  tested  at  the  track  level  since  the  octal  numbers 
(in  the  39th  through  41st  columns),  375,  777,  775,  377,  335,  001  appear  to 
comprise  all  tracks,  with  no  new  occurrences  past  the  36th  run  number. 

As  noted  above  the  number  of  runs  made  could  not  serve  to  test  the 
entire  (113  segments)  program.  But  it  is  interesting  to  analyze  the  problem 
from  the  bottom  up. 
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Table  I  (Part  1).  ORLA  Segment  Usage  Versus  Trial  Number 
(Page  1  of  2) 

QLOOP  :  100  | 

1  :  1  575677777777777777577777617755660757673750000000 

2  S  1  775757777777773777777777775740000157667770000000 
3:1  773677777777773777777777777757000700077750000000 

4  :  1  573677777777773777577737617753760700063770000000 

5  :  1  777777777777777777777777777740000100067750000000 

6  !  1  571677777577773777577737612751660757663350000000 

7  :  1  777677777777773776177000007753000700067770000000 

8  :  1  577677777177737777577777617751000700063750000000 

9  :  1  571777777777773777777777736740000155663350000000 

10  :  1  775777777777773777577777617740000157667770000000 

11  :  1  575777777777773777777777777740000157663350000000 

12  :  1  577357777757703736177000003640000100063750000000 

13  :  1  771677777177737777377777537755000757777750000000 

14  :  1  575677777777733777777777777757765757673750000000 

15  :  1  571657777777733776177000005753000755663750000000 

16  :  1  575777777577777776177000007740000155663750000000 

17  !  1  575777777777743777777777777740000157663750000000 

18  :  l  771677777777777777577777617755000757677750000000 

19  :  1  577777777777773777777777777740000100063750000000 

20  S  1  571677777577773777401777755015000757673750000000 

21  :  0  777777777777777777777777777740000100067750000000 

22  :  1  571777777777743777777777757740000157663750000000 

23  S  1  776263777577743776177000002653000700067750000000 

24  :  1  575657777177733776177000003751000757663750000000 

25  5  1  577657777577773777577777617753000700063750000000 

26  :  1  575777777777773776177000007740000157663750000000 

27  !  1  776277777577743777401777614653000700067350000000 

28  !  1  573677777577773777777777773757000700073750000000 

29  :  1  774777777577773776177000003740000157667750000000 

30  !  1  571677777777773776177000007757000757673750000000 

31  S  1  777777777577767777777777777740000100067750000000 

32  :  1  777677777777767777577777617751000700067750000000 

33  :  1  575677777577767777577777617753000757663750000000 

34  :  1  573677777777733776177000006753765700063750000000 

35  :  1  577677777777733776177000002753760700063750000000 

36  !  1  570677777777743777777777773753760757660010000000 

37  :  1  771777777777777777577777617740000157667750000000 

38  :  1  771677777777737777777777773755000757677350000000 

39  S  1  577677777777773777777777757755000700073750000000 

40  :  1  573757777577767777577777617740000100063350000000 

41  :  1  77367777777777 J7775 77777 6 17755 00070007775 0000000 

42  :  1  573677777777773777577777757751065700063750000000 

43  :  1  7717777775777437775777775577*0000157667750000000 

44  :  l  775677777577777777777777737755000757677750000000 

45  :  1  577677777777733776177000002751000700063350000000 

46  S  1  775777777177733776177000007740000157667750000000 

47  :  1  575677777777777777777777777757000757673750000000 
49  :  1  570777777777733777777777777740000157660010000000 

49  I  1  573677777777733777777777777751065700063350000000 

50  !  1  777677777757703736177000003751000700067350000000 
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Table  I  (Part  2).  ORLA  Segment  Usage  Versus  Trial  Number 
(Page  2  of  2) 


51  S  1  773677777177733776177000007753000700067750000000 

52  :  1  573677777777737777777777777755660700073350000000 

53  :  1  771677777777737777777777777755000757677350000000 

54  :  1  775777777777733776177000007740000157667750000000 

55  :  l  575677777777737777577777617751000757663750000000 

56  :  1  771777777577773776177000007740000157667350000000 

57  s  1  571677777777773777777777777753000757663750000000 

58  S  1  775777777777777777577777617740000155667750000000 

59  :  1  773677777777733776177000007757065700077350000000 

60  :  1  773677777577773777777777777751660700067750000000 

61  S  1  771777777777773777377777415740000157667750000000 

62  S  1  775663777777733776177000002757000757677750000000 

63  :  1  773677777177733776177000003751000700067350000000 

64  S  1  571677777177733776177000007757065757673750000000 

65  S  1  573677777777767777777777773757760700073350000000 

66  S  1  573677777577743777577777617755000700073750000000 

67  :  1  573677777777773777777777777755000700073750000000 

68  :  1  77377777757777377757773761774000010006775000 0000 

69  :  1  777677777777773777577777617751000700067750000000 

70  S  1  577677777777777777577777617755000700073750000000 

71  :  1  571677777577777777577777617755000757673750000000 

72  :  1  771777777777773777777777777740000157667750000000 

73  S  1  777677777177737777777777777755000700077750000000 

74  !  1  575677777777773777577777617757660757673350000000 

75  !  0  575677777777773777577777617757660757673350000000 

76  :  1  575677777777777777777777737755065757673750000000 

77  J  1  571677777777743777777777777753000757663750000000 

78  5  1  577677777577773777577737617757760700073750000000 

79  :  1  7757777777777737761 77000007740000157667750000000 

80  !  1  775777777777773777777777777740000157667750000000 

81  S  1  777777777577777777777777733740000100067750000000 

82  J  1  777777777777737776177000006740000100067750000000 

83  :  1  575777777577777777577777617740000157663750000000 

84  :  1  571677777777743777777777777757000757673750000000 

85  :  1  777673777777733776177000007757000700077750000000 

86  :  1  577777777177733777777777775740000100063750000000 

87  s  1  775677777777733777377777417757765757677750000000 

88  s  1  573677777777733776177000007755660700073750000000 

89  :  1  577677777777773776177000003751000700063750000000 

90  s  1  775777777577773777577737617740000157667750000000 

91  S  1  577777777777777777777777777740000100063750000000 

92  !  1  775777777777733776177000003740000155667750000000 

93  !  1  771677777777773776177000007757165757777750000000 

94  :  1  773677777577773777777777777755065700077750000000 

95  :  1  773677777577767776177000003753000700067750000000 

96  :  1  777777777777773777777777757740000100067750000000 

97  :  1  777777777577777777777777617740000100067750000000 

98  S  1  775677777177727776177000007753000757667750000000 

99  :  1  573777777777777777777777617740000100063750000000 
100  :  1  777677777577773777777777777751000700067750000000 
ALL  S  0  777777777777777777777777777757765757777770000000 


rV 

MCOOWMffU  OOUOL44 


71 


The  initial  analysis  on  the  main  program  was  carried  out  on  the  first  five 
octal  numbers  (representing  the  15  initial  segments  of  the  list  of  segments 
shown  in  Appendix  A).  It  is  well  to  note  again  that  the  octal  number  57567, 
or  binary  101111101110111  associated  with  the  first  run,  means  that  segments 
2,  8,  and  12  were  not  exercised  and  all  the  rest  were  exercised.  By  comparison 
of  the  first  five  numbers  of  each  run  with  its  predecessors  the  pattern  of 
occurrences  of  new  (partial)  tracks  can  be  established.  From  this  sequence 
the  X.j's  of  the  algorithm  can  be  established  as 


X1  “  X2  *”  ~  X12  ~  X1 3 


14  = 

2, 

X 

cn 

ii 

4, 

II 

X 

4 

17  = 

X18  1; 

X19  ~  2> 

X20 

21 

7, 

X22 

1, 

X23 

3, 

24  = 

4, 

X25  = 

4, 

X26  = 

14 

27  = 

6, 

X 

ro 

00 

ii 

31  . 

(The  Sequence  of  Boolean  symbols  is  not  written  because  they  can  be  recovered 
from  the  X^:  13  ones,  1  zero,  1  one,  3  zeroes,  etc.) 

The  ratio  £(i-l)  X^/eX^  in  this  case  is  20.94,  and  the  tables  in 
Appendix  II  of  Reference  1,  indicate  that  the  expected  residual  track  count 
(by  extrapolation)  is  less  than  0.04  (notwithstanding  the  occurrence  of  a 
unique  track  on  the  99th  run). 

For  the  first  30  segments  (i.e.,  first  10  octal  numbers)  the  pattern  is 


1 

X2 

—  •  •  • 

=  X13  =  1 

;  X14  = 

:  2’  X15  =  1;  X16  =  2 » 

17  = 
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4,  Xig  = 

X20  = 

X21  =  X22  =  X23  =  X24 

25  = 
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X26  = 

cn 

CM 

X 

II 

CO 

C\J 

X 

II 

IX 

CM 

X 

i  =  1 1  X30  =  3;  X31 

32  : 

=  X33  *  2; 

X34  ~  3; 

X35  " 

9;  X36  =  X37  =  2; 

38  ; 
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X39 

X40  =  5; 

X41  = 

7;  X42  X43  1  ’ 

44  : 

=  3; 

X45 

8;  x46  - 

X47  = 

1 
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The  ratio  s ( i -1 )  X ^ / X ^  is  29.13  and  tables  for  n  =  47  in  Appendix  A  of 
Reference  1,  show  that  this  corresponds  to  a  residual  count  of  10.9  tracks. 

In  the  context  of  the  present  illustration  this  means  that  there  remain 
10.9  tracks  which  will  exercise  the  first  30  segments  differently  from  the 
way  they  were  exercised  in  the  first  100  runs  and  which  will  differ  from 
one  another. 

A  very  coarse  approximation  to  the  total  testing  required  can  be  found 
by  multiplying  the  number  of  remaining  tracks  by  the  mean  number  of  trials 
between  occurrences  of  the  next  track  as  provided  by  the  entry  for  the  MTTF 
analog  to  this  in  the  tables  of  Reference  1.  In  this  case  this  meantime  is 
about  5.8  so  that  total  testing  will  require  in  excess  of  63  additional  trials 
(163  in  all). 

A  better  estimate  can  be  obtained  repeated  use  of  the  tables,  in  this  way 
the  stretching  out  of  the  MTTF  which  occurs  as  new  tracks  are  found  can  be 
accounted  for.  Using  the  aforementioned  tables  this  estimate  for  the  addi¬ 
tional  trials  s  is 

s  *  5.6  +  5.7  +  6.0  +  6.5  +  7.0  +  7.7  +  8.5  +  9.9  +  12.0  +  16.6  +  30.1 
=  115.6 

where  the  individual  terms  are  taken  from  the  MTTF  column  of  the  tables  for 
n  =  47  to  57.  The  refined  estimate  is  that  about  116  additional  runs  are 
required. 

For  the  45  initial  segments  the  separations  between  new  tracks  are: 

***  ”  ^19  ~  ^ ’  ^20  ~  ^21  -  ^22  ”  ^23  =  ^ ♦ 

X24  =  2;  X25  =  X26  =  =  X37  =  1;  X38  =  3; 

X39  =  1;  X40  =  2  =  X41  =  X42;  X43  =  1  =  X44; 

X45  =  2;  X46  =  X47  =  1;  X48  =  2;  X49  =  X50  =  X51  =  X52  =  1 ’ 

X53  =  2;  X54  =  1  X55;  X56  =  X57  =  2;  X58  =  1; 
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59  =  2;  X60  =  1;  X61  3;  X62  =  1;  X63  3;  X64  ~  1; 


65  3;  X66  =  X67 

70  =  2;  X71  =  2;  X72 


1 ;  Xg8  -  3;  Xgg  1 ; 

1;  X73  =  2;  X?4  x75  =  1 • 


The  pattern  of  separations  of  occurrences,  produces  an  estimate  of  95.5 
(170.5  total)  additional  tracks,  and  a  mean  time  to  next  new  track  of  only  1.82. 
The  number  of  trials  required  to  achieve  perfection  can  be  approximated  by 
the  formula  for  MTTP  for  Section  2. 2. 3. 2.  In  this  case  N  =  170.5,  $  =  0.00576, 
and 


MTTP 


i=75 


iri 

<p  C—j  k 


which  can  be  approximated  by  the  sum  of  the  logarithm  of  n  and  the  Euler 
constant 


MTTP  *  1  (In  95  +  0.57721) 

<p 

=  890 

Thus  890  additional  tests  are  estimated  to  be  required  for  a  complete  test. 

For  the  first  60  segments,  the  pattern  of  87  X^'s,  produces  an  estimate 
of  the  undiscovered  tracks  of  about  359,  of  <f>  =  0.00217,  and  the  total  number 
of  runs  required  for  a  fully  tested  program  is  about 


445 

359 

V  1 

11 

-e-'  — ■* 

M 

Z-f  446  -  i 

i  =87 

k=l 

*  300 

The  analysis  for  the  initial  75  segments  produces  91  separation  intervals, 
with  a  pattern  showing  only  8  values  of  differing  from  1.  These  are  all  2 
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and  occur  at  the  indices  21,  43,  51,  60,  71,  79,  82,  85,  and  86.  These 
produce  an  estimate  of  N  =  1108,  <t>  =  0.000857.  Corresponding  is  an 


1107  1016 

}  L  TOT-tT  =  I  T.  I 

1=92  k=l 

=  8750 

Naturally  these  later  estimates  are  extremely  weak  with  unquestionably 
extremely  large  variances.  The  point  with  any  such  estimates  is  one  of 
determining  the  status  of  testing  and  gross  estimates  are  sufficient. 

The  preceding  sequence  of  tests  clearly  indicate  by  the  increasingly 
large  value  of  MTTP  that  the  testing  required  is  extremely  large,  probably 
in  excess  of  50,000  runs.  And  this  is  for  track  level  testing  on  coverage, 
not  execution  sequence  coverage. 

It  is  useful  to  note  that  the  bottom  row,  denoted  all,  which  shows  in 
each  position  the  "union"  of  all  octal  tokens  above  it  in  the  column,  shows 
segment  coverage  complete  through  85  segments.  (86  was  noted  before  an 
impossible  segment.) 

3.2.2  Comprehensive  Testing  of  Matrix  Triangul ization  Problem 

The  matrix  triangularization  example  discussed  earlier  will  be  reexamined. 
Directed  graphs  of  the  potential  program  flow,  and  a  few  examples  of  the 
coverage  by  random  numbers  and  constructed  cases  were  given  in  Section  3.1.2. 
Listings  of  the  MAIN  and  TRIANGULARIZATION  subroutine  comprise  Figure  12. 

Appendix  B  contains  tables  and  reports  of  the  APTS  output  for  three 
separate  test  runs.  The  reports  show  the  testing  coverage  provided  by  using 
the  user-described  input  routine  INR0UT.  INR0UT  returns  a  new  set  of  randomly 
distributed  over  the  logarithm  in  the  range  -2  to  1 .  The  sign  of  the  individual 
data  items  is  also  selected  randomly. 
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Figure  12.  Lilting  of  an  Example  Program  (Page  1  of  2) 
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Figura  12.  Listing  of  an  Exampia  Program  (Paga  2  of  2} 


The  first  three  cases  of  test  run  No.  1  (see  Pages  B-2  to  B-4  in  Appendix  B) 
show  coverage  of  code  for  the  MAIN  program  as  100%  in  the  column  marked  Summary. 
Subroutine  TRIAN6  gets  a  summary  coverage  of  86.96%.  The  remaining  segments 
to  be  tested  are  numbers  3,  12,  and  16,  as  seen  in  the  segment  reference  report 
(Page  B-3  of  Appendix  B).  The  segment  reference  tables  are  used  to  relate  the 
segment  numbers  and  their  corresponding  program  statement  numbers  together. 

As  an  example,  it  is  seen  that  segment  3  contains  lines  34,  35,  36,  and  37  in 
subroutine  TRIANG  (see  Figure  12).  These  lines  correspond  to: 

IF(A(K,K).EQ.O)34  I P ( N )  =  035 
CONTINUE  -+  K  =  K+l 36  IF  (K.LE.N)37  loop 

(DO-loop  termination  includes  an  implied  conditional  branch) 

Following  the  summary  reports  and  the  segment  reference  tables,  the  trial 
statistics  appear  on  Page  B-4,  for  example.  These  are  the  that  are  needed 
to  calculate  the  estimate  of  the  number  of  remaining  tracks.  (Actually,  more 
than  three  cases  are  required  for  the  estimation  and  the  three  entries  on 
Page  B-4  form  only  a  part  of  the  data  used.) 

Supplied  as  part  of  the  testing  package  is  a  program  that  interacts  with 
the  user  and  calculates  the  difference  of  the  two  sides  of  the  estimation 
equation  in  Paragraph  3.1.4  based  on  trial  solutions  supplied  by  the  user. 

To  explain  how  the  X..  are  formed,  the  formation  of  X-j  and  X^  will  be 
considered.  Case  1  of  run  1  (see  Page  B-2)  shows  the  number  of  times  each 
segment  of  MAIN  and  TRIANG  were  executed.  Since  this  is  the  first  test  case, 
the  first  unique  track  is  automatically  formed.  Case  2  of  run  1  for  TRIANG 
(Page  B-2)  shows  the  same  segments  being  executed  (the  number  of  executions 
of  each  segment  listed  happen  to  be  the  same,  but  this  is  irrelevant,  the 
comparison  is  made  on  the  basis  of  whether  or  not  the  segment  was  executed, 
not  on  how  many  times)  as  in  case  1,  run  1.  However,  the  MAIN  routine  shows 
a  difference  in  execution.  Therefore  case  1  and  case  2  are  different,  so  we 
form  X^  =  1 .  This  means  that  one  case  occurred  since  the  last  unique  track. 

If  we  compare  case  3  of  run  1  against  cases  1  and  2  we  also  find  a  difference 
in  the  MAIN  routine  (see  segment  3  execution  counts).  This  gives  us  our 
third  unique  track.  Hence,  =  1*  also,  since  only  1  case  occurred  since 
the  last  unique  track. 
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Continuing  in  this  fashion,  by  comparing  cases  1  through  9  (in  Appendix  B), 
in  order,  we  find  unique  tracks  for  cases  1,  2,  3,  4,  and  8.  (A  summary  of 
the  nine  cases  is  found  on  Page  B-10.)  The  Boolean  tokens  associated  with  the 
sequence  are  shown  below: 


By  using  the  estimation  equation,  it  was  determined  that  there  existed  9.1  new 
tracks  to  be  found. 

Appendix  C  contains  reference  tables  of  the  APTS  output  for  a  constructed 
case.  The  constructed  case  shows  the  use  of  monitor  variables  (page  C-2). 

For  constructed  cases,  the  user  is  required  to  supply  input  data  to  the  program, 
and  to  supply  the  monitor  variables.  It  is  seen  that  the  user-supplied  input 
is  in  the  DATA  statement  in  the  MAIN  program.  Subroutine  TRIANG  shows  the 
use  of  monitors  inserted  into  the  program  of  branch  points. 

By  analyzing  the  unexercised  segments,  3,  12,  and  16  of  the  three  test 
runs  of  Appendix  B,  where  they  are  marked  by  asterisks  in  all  three  of  the 
segment  reference  tables  (Pages  B-3,  B-6,  B-9),  it  can  be  determined  from  the 
listing  that  the  variable  T  holds  the  key  to  exercising  these  segments. 

Further  examination  suggests  that  if  A[3,3]  is  equal  to  zero  then  segment  3 
will  be  exercised. 


Segment  12  requires  variable  T  to  be  zero.  For  this  to  be  true,  A[l,l] 
could  be  equal  to  zero  or  |A[3,2][  could  be  greater  than  |A[2,2]|  and  A[l,2] 
must  be  equal  to  zero. 

Segment  16  also  requires  variable  T  to  be  equal  to  zero.  This  condition 
will  result  if  A[l,l]  is  not  zero  and  A[l,2]  is  equal  to  zero. 


These  findings  determined  the  initial  values  of  A  for  the  DATA  statement. 
By  observing  the  segment  reference  for  subroutine  TRIANG,  we  find  that  seg¬ 
ments  3,  12,  and  16  have  been  executed  and  the  test  coverage  is  complete. 
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3.3  ADDITIONAL  PROBLEMS  IN  COVERAGE  TESTING 


3.3.1  Formation  of  Execution  Sequences 

It  is  well  to  state  at  the  outset  that  only  the  outline  of  this  problem 
has  been  established.  The  following  paragraphs  describe  the  background  and 
outline  of  the  problem. 

The  use  of  tracks  as  proxies  for  execution  sequences  is  in  part  necessary 
and  in  part  expedient.  Tracks  are  necessary  because  one  usually  cannot  deter¬ 
mine  the  actual  sequence  from  a  list  of  usages:  with  several  entrances  and 
several  exits  from  a  node  and  a  different  usage  number  of  each,  there  is 
usually  no  way  to  determine  the  actual  sequence  of  the  computation  that  would 
produce  the  usage  numbers.  On  the  other  hand,  information  often  is  available 
which  would  allow  the  program  flow  to  be  determined  in  a  gross  or  general  way, 
and  that  information  heretofore  has  not  been  employed  in  our  studies.  It 
would  be  helpful  to  program  testers  to  provide  a  general  sequence  of  the  flow 
resulting  from  a  given  input  driver  set. 

To  illustrate  this,  an  example,  depicted  in  Figure  13,  shows  the  set  of 
executed  segments  and  their  counts  as  solid  lines  or  arcs  between  all  nodes 
which  were  passed  during  the  first  data  set  employed  on  the  ORLA  program. 

It  will  be  noted  that  dotted  lines  are  also  shown  emanating  from  certain  of 
the  nodes  which  were  passed.  These  are  branches  which  were  not  taken  on  this 
run;  they  would  be  important  in  coverage  testing  but  can  be  ignored  for  the 
present  discussion.  The  flow  of  the  computations  can  be  determined  unambigu¬ 
ously  only  in  the  cases  where  a  single  execution  is  performed  on  a  segment 
and  no  other  segment  parallels  the  segment.  For  example,  there  is  such  a 
segment  joining  nodes  355  and  263  in  the  central  lower  one-third  of  the  chart. 
This  and  others  are  highlighted  in  Figure  14,  where  they  may  be  easier  to 
1 oca  te . 

The  general  flow  can  be  formed  from  the  unambiguous  segments  which  show 
a  usage  of  one.  In  one  case,  there  are  (at  node  226)  two  segments,  both  with 
a  count  of  1,  shown  exiting  a  node.  But  this  particular  ambiguous  case  is 
easily  resolvable  (i.e.,  precedence  determined)  because  the  branch  along 
segment  13,  joining  nodes  226  and  483,  joins  to  the  exit  (END),  and  so  cannot 
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precede  the  segment  joining  nodes  226  and  235.  This  suggests  an  interesting 
problem  of  which  the  preceding  example  is  the  most  trivial:  given  a  set  of 
nodes  and  their  counts,  determine  under  what  conditions  the  actual  flow  can 
be  determined.  This  "academic"  problem  will  not  be  pursued  in  this  study. 

The  application  of  the  simple  rule  which  establishes  the  one-time  and 
segments  (a  "footprint,"  or  better,  a  "one-print")  permits  a  linking  of 
certain  segments  to  form  contiguous  blocks  of  the  program,  the  General  Flow 
of  the  title  of  Figure  14. 

Such  linkings  are  shown  in  Figure  14,  where  the  defined  flow  consists  of 
the  following: 

Block  1:  Segments  1,  113,  4,  6,  112,  8,  102,  11,  87,  101 

Block  2:  Segments  96,  18,  86 

Block  3:  Segment  26 

Block  4:  Segments  30,  32,  33,  45,  36 

Block  5:  Segments  41,  42,  13,  15,  17  (END) 

Even  the  undefined  flow  can  be  combined  to  form  pseudo  segments  if  there 
are  not  dotted  lines:  thus,  the  series/parallel  segments  20,  21,  84,  22,  23, 
24,  and  25,  which  are  between  nodes  319  and  355,  can  be  treated  as  a  single 
pseudo  segment  with  a  usage  of  150,  the  entry  and  exit  counts  at  the  two 

joined  nodes.  In  addition  to  these  pseduo  segments,  another  type  of  merging 

is  possible  in  certain  areas.  For  example,  some  of  the  segments  from  Block  4 
of  the  above  list  can  be  joined  with  the  segment  of  Block  3  to  form  a  super¬ 
block.  Since  all  possible  paths  to  and  from  nodes  263  and  273  have  been 
exercised,  these  can  be  eliminated  from  further  consideration,  permitting 
formation  of  a  pseudo  segment  with  which  to  join  segment  30  to  segment  26. 

Also,  since  node  291  has  all  exits  exercised,  it  too  can  joint  to  form  a 
larger  block  (26,  pseudo  segment,  30,  and  32).  Because  node  292  has  a  dotted 
line  out  of  it,  there  is  no  further  merging  possible  between  the  two  blocks. 

Even  though  the  remainder  of  the  program  flow  is  undefined,  there  are 
many  points  which  are  internal  to  the  undefined  blocks  where  reduction  is 
possible.  A  trivial  example  is  the  pair  of  parallel  paths  91  and  99  between 
nodes  191  and  209,  which  can  be  merged  into  a  two-use  segment;  more  interesting 
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cases  can  be  identified  in  the  lower  left  portion  of  Figure  13.  Thus,  between 
nodes  401  and  415  are  segments  53,  68,  54,  55,  57,  56,  and  58,  all  of  which 
can  be  merged  to  a  118-use  pseudo  segment. 

Figure  15  shows  a  considerably  pruned  version  of  the  flow  diagram.  As 
with  the  preceding,  it  is  developed  from  the  one-prints  and  more  is  required 
to  establish  the  sequence.  For  example,  segment  42  appears  to  follow 
(dynamically)  41,  but  there  is  no  reason  to  think,  a  priori,  or  in  a  local 
context  that  it  actually  does.  In  a  global  context,  however,  it  is  known 
that  segment  42  is  the  later  exit  from  node  385,  because  42  joins  to  226  and 
from  there  out  to  the  END. 

The  primary  purpose  of  investigation  of  the  problem  of  pruning  the  flow 
diagram  was  to  assist  in  the  development  of  a  display-aided  test  bed,  where 
sections  of  the  program  could  be  showed  in  network  form  and  successively 
pruned  on  a  case  by  case  basis. 

3.3.2  Partially  Automated  Test  Bed 

In  keeping  with  the  desire  to  achieve  economical  testing,  the  goal  was 
to  automate  the  entire  process  which  has  been  outlined  in  the  preceding 
discussion. 

The  major  problems  in  completely  automating  the  cover-testing  process  are 
in  construction  of  the  software  required  to  establish  the  status  of  testing, 
maintain  suspense  files  on  all  unexercised  program  segments,  insert  augmenting 
viarables  corresponding  to  predicates  which  define  the  entry  into  the  (unex¬ 
ercised)  computational  segments,  search  the  input  variable  space  to  achieve 
entry,  compare  the  resulting  track  with  previously  obtained  tracks,  and  prune 
the  original  tracks  to  a  set  of  smaller  dimension  (manifested  in  the  reduction 
of  the  original  n-tuples  to  tuples  of  smaller  size).  This  is  to  be  done  within 
the  restricted  physical  environment  of  current  displays  and  I/O  equipment. 

The  complete  list  of  tasks  required  is  briefly  sunmarized  once  again: 

A.  Identify  unexercised  branches  (at  the  end  of  the  initial  runs  with 
random  numbers). 
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B.  Pick  an  unexercised  branch  and  display  the  listing  associated  with 
the  branch  (a  "back"  sort  is  required  which  identifies  the  instruction  number 
of  the  involved  predicate). 

C.  Formation  of  an  auxiliary  variable  based  on  the  nature  of  the 
predicate.  (For  example,  if  the  test,  A<B,  is  the  predicate,  the  auxiliary 
variable  could  be  =  B-A). 

0.  Create  a  variable  (with  requisite  modifications  to  the  object 
program).  Recompile  the  program. 

E.  Vary  input  variables  until  the  auxiliary  variable  is  positive. 
Rationale  for  the  variation  depends  on  the  program  variables  identified  in 
the  listing. 

F.  List  all  exercised  segments  and  compare  with  preceding  usage. 

G.  "Release"  the  variable  and  proceed  to  a  new  unexercised  branch. 

H.  In  an  extension  of  the  above  procedure,  several  auxiliary  variables 
can  be  inserted  at  one  time  and  input  data  chosen  in  some  systematic  way 

(a  search)  to  achieve  arbitrary  valuations  on  all  auxiliary  variables. 

The  results  of  a  run  or  series  of  runs  can  be  displayed  in  the  form  of 
a  list  of  unexercised  segments.  It  is  clear  that  the  information  of  the  type 
shown  in  the  bottom  row  of  Table  1,  can  provide  a  quick  look  at  the  status 
of  segment  coverage  after  an  initial  set  of  runs  has  been  made.  The  segment 
numbers  marked  by  asterisk  as,  for  example,  on  Page  B-3  of  Appendix  B,  can  be 
displayed. 

The  back  sort  to  identify  the  instruction  number  at  the  beginning  of 
any  chosen  segment  can  be  easily  automated. 

The  process  of  inserting  auxiliary  variables  at  the  predicates  associated 
with  unexercised  segments  at  present  must  be  done  manually.  The  problem  of 
inserting  the  variables  requires  a  recompilation  and  this  must  be  monitored. 
Development  of  the  form  or  expression  with  which  to  represent  the  auxiliary 
variable  may  require  scanning  the  listing  over  an  extensive  set  of  instructions. 


MCDOMWtU  DOI/OLA9 


86 


Section  4 

ERROR-DETECTION  MODELS 


4.1  SUMMARY 

Two  variations  of  the  Jelinski-Moranda  model  were  developed  for  estima¬ 
tion  during  program  development.  The  first  permits  estimation  of  the  error 
content  of  the  completed  software  package  using  data  which  is  taken  on  only 
portions  of  the  package.  That  model  is  applicable  when  the  eventual  size  of 
the  program  is  known  at  the  outset. 

The  second  model  permits  a  similar  analysis  during  the  development  of  any 
software  package  which  is  homogeneous  with  respect  to  its  complexity  (error 
making  and  finding). 

These  models  should  assist  analysts  in  an  early  determination  of  error 
content.  They  should  also  eliminate  the  present  practice  of  applying  models 
to  the  wrong  regime  (decreasing  failure  rate  models  applied  to  growing-in¬ 
size  software). 

4.2  INTRODUCTION 

In  normal  usage  of  the  Jelinski-Moranda  model,  the  software  package 
under  test  is  assumed  to  be  of  fixed  size  with  a  fixed  number  of  incipient 
errors.  The  size  of  the  package  does  not  appear  explicitly  in  the  model  as  a 
parameter,  and  its  effect  is  only  indirectly  realized  by  the  way  it  affects 
the  number  of  incipient  errors  which  exist  at  the  start  of  testing  (there  is 
a  direct  relation  between  instruction  count  and  error  count). 

The  model  could  not  be  employed  legitimately  on  software  packages  which 
were  incomplete.  Several  workers  attempted  to  fit  the  model  to  an  initial 
period  of  time  when  its  error  rate  was,  indeed,  increasing,  due  to  the  grow¬ 
ing  size,  and  they  met  with  no  success.  (As  a  matter  of  fact,  the  only  models 
which  produced  reasonable  estimates  when  applied  during  this  regime,  were 
the  Increasing  failure  rate  models.) 
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It  would  be  helpful  if,  at  the  outset,  an  estimate  could  be  obtained  of 
the  total  error  count  which  will  be  realized  in  test  and  usage  of  a  package. 

Recent  work  by  IBM  (Reference  25)  has  prompted  a  reexamination  of  the 
original  Jelinski-Moranda  model  for  the  purpose  of  incorporating  the  (changing) 
program  size.  This  turns  out  to  be  very  easily  effected  if  good  record  keep¬ 
ing  can  be  maintained  during  program  development  so  that  the  size  of  the 
package  is  recorded  as  a  function  of  some  convenient  timing  metric  (CPU  or 
calendar).  Following  is  a  description  of  the  analysis. 

The  original  model  is  depicted  in  Figure  16,  where  the  two  parameters 
are  shown  in  Figure  18(b),  and  a  typical  realization  of  the  error-finding 
process  is  shown  in  Figure  16(a).  N  is  the  initial  error  content  (of  a  com¬ 
pleted  program)  and  4>  is  the  contribution  to  the  error  rate  due  to  a  single 
error. 

While  the  meaning  of  4>  is  maintained  in  the  two  models,  the  meaning  of 
"initial  error  content"  needs  to  be  clarified.  This  is  done  below  in  the 
description  of  Model  1,  where,  in  effect,  N  maintains  its  meaning  as  the 
number  of  errors  in  a  completed  package.  In  the  second  model,  a  fixed  error 
rate  per  instruction  is  assumed,  and  growth  of  the  package  is  measured  by  the 
count  of  instructions  (under  test)  versus  time. 

4.2.1  Model  1 

Let  S(t)  denote  the  fraction  of  the  total  number  of  statements  which  a 
complete  program  will  have.  The  metric  t  is  measured  in  terms  either  of  the 
accumulated  CPU  time,  or  of  the  amount  of  calendar  time,  which  has  been  used 
for  testing  the  package. 

The  simplest  way  of  introducing  the  effect  is  to  use  S(t)  as  a  "modulation" 
of  the  error  detection  rate  Z(t)  of  the  original  model.  In  the  notation 
formerly  employed,  this  combined  or  modulated  rate,  denoted  W(t),  is: 

W(t)  =  S(t)  Z(t) 

=  s(t)  [(N-i+1  )4>D  for  T!^  <  t  <  T1  (2) 
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and  Tj,  T£,  T^,...,  denote  tho  times  of  detection  for  the  errors.  (Primes 
are  employed  on  T's  to  distinguish  them  from  the  times  of  the  original  process.) 
The  effect  of  S(t)  on  the  T!  should  be  made  clear  at  the  outset.  When  S(t), 
the  fraction  of  the  total  count,  increases,  the  composite  error  rate  will 
generally  increase,  as  will  the  liability  for  error  for  the  "modulated" 
process.  For  this  reason,  the  times  T!  for  the  composite  process,  W(t),  will 
be  different  from  the  of  the  Z(t)  process.  Since  N  in  the  original  model 
represented  the  total  error  content  of  a  complete  software  package,  a  proper 
correspondence  which  preserves  the  meaning  is  that  N  is  the  error  content  at 
a  time  corresponding  to  the  completion  of  the  software  package,  S(t)  =  1.00. 

This  necessarily  presumes  that  the  size  of  the  package  which  will  be  developed 
is  known  at  the  outset.  (SQ(t)  would  represent  the  fraction  of  the  total 
which  is  accomplished  at  time  t.)  This  may  or  may  not  be  a  serious  barrier. 

Some  modules  can  be  sized  at  the  outset,  but  large  complex  programs  may  not  be. 
An  alternative  to  this  is  offered  subsequently  in  Model  2. 

For  the  present  model,  SQ(t)  is  a  nondecreasing  function  which  starts  at 
zero  at  Tq  and  achieves  its  maximum  value  at  some  unknown-at-the-outset  time. 


Thus,  0  <_  SQ(t)  £  1,  with  s0(Tq)  =  0  and  SQ(Tp  =  1. 

While  SQ(t)  is,  in  the  large  sense,  random,  the  records  of  progress  will 
permit  specific  values  of  So( t )  to  be  determined  and  the  randomness  is  of  no 
concern.  In  particular,  it  is  necessary  that  SQ(t)  can  be  determined  at  the 
epoch  times  Tj,  T^,...,  T^  at  which  the  errors  are  detected. 

When  the  completion  time,  T^,  is  reached  and  for  times  thereafter,  the 
software  package  is  complete  (SqCT^)  =  1)  and,  formally,  the  density  given  in 
Equation  (1)  is  the  same  as  that  given  in  the  original  paper  (Reference  3). 

It  has  been  mentioned  earlier  that  the  time  pattern  of  errors  will  be 
different  for  the  "modulated"  process,  and  it  is  interesting  to  see  just  what 
would  happen  if  Sq(Tq),  or  for  short,  SQ(0),  were  0.10  (10%  of  the  package  is 
initially  available  for  test),  and  it  did  not  increase  beyond  that  for  a  long 
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period  of  testing.  The  time  pattern  of  errors  Tj,  T^,...,  which  would 
occur,  would  have  associated  separations  Xj  =  -  Tq,  Xj  =  ~  Tj,  .... 


X*  =  Tn 

n  n 


T;-r 


Because  SQ(0)  =  0.1 0,  the  composite  detection  rate  for  the  first  error 
would  be  (0.10)  N<)>,  that  is,  10%  of  the  original  error-detection  rate.  This 
means  that  the  first  detection  time  Tj,  would  (on  the  average)  be  10  times 
as  long  as  the  time  for  the  corresponding  error  of  the  unmodulated  process. 
The  second  error  would  have  the  same  property  (on  the  average),  and  so  forth. 
The  implications  of  this  fact  can  be  seen  from  the  following.  The  likelihood 
function  would  be 


L(Xj,  X',  ...,  x;)  =  [1  S  (0)4>[N-(i-l)  exp  {-[S  (OMN-1+1  )X']}. 

i=l 

The  likelihood  equations  obtained  by  differentiating  the  logarithm  of  the 
likelihood  with  respect  to  N  and  <f>  are: 


S,  N-(i-l)  '  So^  *  X1 


J-So(0)  Z  [N-(i-l )]  X] 


As  noted  above,  the  observables  X!  would  be  (about  or  on  average)  10 
times  as  large  as  before.  Thus,  from  Equation  (3b),  the  solution  <j>  will  be 
(on  average)  the  same  as  its  value  for  the  unmodulated  process,  or  for  the 
completed  software  package. 

A 

Using  the  solved-for  value  of  <P  in  Equation  (3a)  and  the  fact  that 
So(0)X]  in  the  new  process  is  the  same  as  Xi  in  the  original  process,  it  is 
seen  that  the  solution  N  is  also  the  same  as  before. 
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The  analysis  then  shows  that  if  it  is  known  that  a  package  under  test 
represents  (in  all  respects)  a  certain  percentage  of  the  total,  then  the  total 
eventual  error  content  can  be  estimated  by  using  these  slightly  modified 
likelihood  equations. 

The  result  is  encouraging  for  the  outlook  for  success  in  the  following 
simple  generalization  of  the  above  example.  In  this  generalization,  the 
SQ(t)  modulating  function  is  constrained  to  be  constant  during  each  test 
interval.  Using  essentially  the  same  notation  as  before,  the  likelihood 
equations  for  the  generalized  modulated  process  are 


"  *  £  Si-1 


0 


(4a) 


and 

J-  I  s.  -i (N-i+l )Xi  =  0  (4b) 

4  1-1  1 

where  S^_-|  is  the  percentage  completion  achieved  prior  to  the  start  of  the  i— 
interval . 

Solutions  for  the  parameters  can  be  carried  out  as  indicated  above  in 
the  example. 

The  mean-time-to-next  error  MTTF  (n+1  s_t  in  the  present  context)  can  be 
estimated  by  evaluating  the  rate  at  time  T^  and  taking  the  reciprocal  of  it. 

In  the  present  case  (using  a  subscript  on  the  left  side  to  correspond  to  the 
model  number): 

MTTF,  =  - K - , 

'  S ( T ' ) ( N-n ) 0 

fi 

/\  A 

where  N  and  *p  are  solutions  to  the  Maximum  Likelihood  Equations  (MLE's). 
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4.2.2  Model  2 

Let  Ep  denote  a  characteristic  rate  of  error-making  for  the  programmer 
(or  programmer  team)  and  the  program  type.  This  rate  will  be  estimated  by 
application  of  the  model  described  subsequently,  but  there  are  some  useful 
facts  concerning  this  parameter. 

In  1975,  it  was  observed  (Reference  23)  that  there  appears  to  be  a 
"...  'universal1  coding  -  error  rate  ...,"  which  has  a  value  of  about  2  errors 
per  100  instructions  (of  the  language  in  which  the  program  has  been  written). 
This  observations  was  based  primarily  on  the  data  (now  famous)  provided  by 
F.  Akiyama,  but  also  on  earlier  observations  made  by  B.  J.  Hatter,  et.  al. 
Subsequently,  the  validity  of  this  "thermodynamically  stable"  parameter  has 
been  reinforced  by  several  other  studies. 

The  interesting  feature  of  some  of  this  later  data  (Reference  24)  is  that 
the  error  rate  of  two  per  hundred  was  observed  on  programs  which  had  completed 
their  development  and  integration  phases;  they  were  under  test  before  the 
relevant  error  counting  was  initiated.  This  is  surprising  since  the  coding 
error  rate  is  thought  of  as  being  similar  to  a  typist's  miskeying,  and  should 
be  purifiable  by  edit  routines  and  by  code  checking  due  to  early  misstarts  of 
the  program. 

These  features  of  an  hypothesized  entity  are  fortunately  not  used  in  the 
following  analysis. 

The  error  rate  at  any  point  in  the  development  of  a  program  whose  cur¬ 
rent  instruction  count  is  G(t)  is  assumed  to  be  proportional  to  the  current 
error  content 

V(t)  =  <t>[G(t).Ep  -  n(t)]  (5) 

where  n(t)  is  the  accumulated  number  of  error  corrections,  and  Ep  is  the 
per  instruction  error  rate. 

As  before,  if  G(t)  can  only  change  at  error-discovery  epochs,  T^,  T2, 

....  Tn,  and,  if  n(t)  also  has  this  feature,  then  the  rate  has  the  form 
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(6) 


V(t)  =  4>[Gi_1  Ep  -  (i-1 )]  for  Ti_1  <  t<T. 

where  Gi_1  =  G(Ti ) ,  and  n(t)  is  i-1  for  the  interval  starting  at  T._j 

Since  G(t)  is  a  function  or  process  which  takes  place  without  any  apparent 
dependence  on  the  error-finding  process  (except  that  the  error  epochs  are 
assumed  to  be  the  points  of  entry  of  new  code)  it  is  reasonsble  to  assume  that 
the  random  time  separations  between  errors  (X-j,  X 2 ,  Xn)  are  statistically 

independent. 

Under  these  conditions,  the  constant  rate  implies  an  exponential  dis¬ 
tribution  for  the  X.j ,  and  the  likelihood  function  for  n  errors  is: 

L( X-j »  *2,  •••»  Xn) 


£  4>CGi_1  E  -  (i-1)]  exp  {-(^.[G^-,  E  -  (i-1)]} 


i=l 


(7) 


The  MLE's  obtained  by  differentiating  the  logarithm  of  the  likelihood 
function  with  respect  to  cp  and  Ep  are: 


n 

£ 


Gi-i 


z-  c - p — ttot  "  ^  ,  x.  =  o 

i=l  bi-i  hP  u  '>  i=l  11  1 


(8a) 


f  -  Z  [G,.,  Ep  -  (i-1)]  X,  -  0  (8b) 

The  MLE's  are  solved  as  before:  Equation  (8b)  can  be  algebraically 
solved  for  <p;  this  is  substituted  in  Equation  (8a),  and  the  resulting  key 
equation  is  solved  for  Ep  by  trial  and  error. 

It  is  recalled  that  the  desired  performance  parameter  is  Ep,  which  can 
then  be  used  with  either  the  current  (known)  or  projected  (estimated)  instruc¬ 
tion  count  to  determine  the  total  error  content. 
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Estimates  of  the  MTTF  at  any  time  can  be  obtained  by  the  formula 


MTTF-  =  x - - -  (9) 

*[Gn  Ep  -  n] 

4.3  CONCLUSIONS 

The  two  models  presented  in  the  analysis  are  both  very  tractible 
analytically. 

Model  1  would  be  of  use  for  those  programs  whose  eventual  size  is  known 
at  the  outset.  It  requires  that  a  record  of  the  times  of  error  occurrences 
be  maintained  as  well  as  a  record  of  the  percentage  of  completion  at  each  of 
the  error-detection  times.  It  provides,  at  any  stage  of  testing,  an  estimate 
of  the  error  content  of  the  untested  complete  package. 

Model  2  applies  to  any  developing  software  package  which  is  homogeneous 
with  respect  to  the  complexity  of  programming  and  with  respect  to  the  talents 
of  the  programmers.  The  important  property  is  that  Ep,  the  error-making 
rate  (or  error-finding  rate),  must  be  a  constant  across  the  entire  software 
package.  In  case  of  inhomogeneity  separate  analyses  are  advised. 

4.4  GLOSSARY 

Terms  and  symbols  used  in  the  preceding  sections  are  identified  as 

A  "modulation  function"  which  ranges  from 
0  <_  S  (t)  1.0  and  is  nondecreasing.  It  represents 

the  fraction  of  the  code  completed  up  to  time  t.  It 
s  a  given  for  the  problem. 

The  Jel inski -Moranda  detection  or  purification 
process. 

The  product  of  S(t)  and  Z(t).  It  represents  the 
error-making  or  error-detecting  rate  versus  time 
for  Model  1 . 

The  number  of  errors  in  the  completely  coded 
software  package.  This  is  estimated  from  data. 


follows: 

SQ(0) 

Z(t) 

W(t) 

N 
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The  contribution  of  one  error  to  the  detection 
(failure)  rate.  This  is  estimated  from  data. 

The  time  at  which  the  i—  error  is  found,  measured 
in  any  convenient  timing  metric.  An  observable. 

The  separation  between  the  i—  and  the  i-1— 
error.  An  observable. 

The  cumulative  number  of  errors  found  in  testing 
up  to  time  T^. 

Is  the  percentage  of  completion  during  the  (i+1)  st 
interval.  This  is  provided  as  exogenous  data. 

The  estimated  meantime  to  error  obtained  by  using 
Model  i  (i  =  1,2). 

The  nondecreasing  function  representing  the  total 
instruction  count  of  the  package  at  time  t.  This 
is  a  given  for  the  problem. 

The  error  making  rate  for  a  given  program-progranmer 
mix.  It  is  estimated  from  data. 

The  number  of  errors  found  during  test  up  to  time  t. 
An  observable. 

A  stochastic  process  representing  error-making 
or  error-detecting  rate  versus  time. 

The  generic  representation  for  the  likelihood 
function. 
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Appendix  A 

AUGMENTED  ORLA  PROGRAM 


ooono  r>  o  n  oo.Tr».ic->rjrjOc*»nnnorioo"Jor>ooonoor><*> 


ORL  A 


MD  AC  SFSMENT  XLATOR 


10/ 31/19B0  11:36  PM 


PAGE 


1 


PROGRAM  ORLA 

—  OPTIMUM  RFPAIR  LEVEL  ANALYSIS  (OftLA) 

—  O.R.  JOHNSON  ACDCM-WARNSR  RODINS  ALC-5503 


—  THE  CONSTANTS  ARE  ASSIGNED  ACCORDI NGLY- 


1-RRT 

2-CON 

3-DLA 

4-DLWP 

5-OPL 

6-nss 

7-FLA 

8-FLWR 

9-IAC 

10-IPC 

11-K 

1 2-0 ST  ICON 

13-OSTIOS 

14-OSTNCON 

1 5-OSTNOS 

16-PILP 

1 7-PSLRCON 

1 J-PSLROS 

19-PSMRCON 

20-PSMRCIj 

21-PWRCON 

22-PMROS 

23-RAC 

24-RPC 

75-SA 

26 -SR I  CON 

37-SP IDS 

78-SRNCUN 

29-SWNOS 

30-TD 

31-UE 

3  2 -'JR 

33-VD 

34-VF 

35-YT 

36-ZT 

—  THE  VARIABLES  ARE  ASSICNED 

ACCORDINGLY- 

37-DF 

33-UMR 

39-FAF 

40-FF 

41-H 

42-J 

43-LA 

44-LP 

45-PP 

46-GRA 

47-RMfi 

48-SC 

49-S* 

50-UC 

5 1-UV 

52-W 

53-X 

54-YIJ 

55-  GO 

—  THE  COMPUTED  VARIABLES  ARE 

ASSIGNEE  ACCORDINGLY- 

56-MTBC 

57  —  Li  A 

58-DAM 

59-FS 

60-FAM 

—  THE  ACE 

COST  MARIABLES  ARE 

ASSIGNED  ACCORDINGLY- 

6 l-DCS 

62-DUA 

63-TC5 

S4-IUA 

—  STORAGE 

TABLES 

INTEGER 

NVARCI 8) 

z  KF ( 1 1 )  / K D ( 1 2 ) 

/ 

- 

K  *’*  <  "3  3  ,KSEN(6),JSENn)#.NiJMvr(3) 

/ 

NUM1(3)/IPASC 

100) 

REAL 

V  AL( 6  4 ) 

/ F V( 11)  / 0 V(  1  2) 

9 

- 

QCTGV(IOG) 

/T V( 3  )  / 

- 

HRS(IOC) 

/TTTMdOO  ) 

9 

A I  A ( 1 00 ) 

/DAA( 100) 

REAL 

ANSWER  /BLANK 

/DASH  ,NO 

9 

OUT( 3  )  /X 

/YFS 

DQUHLE  PRECISION  AGE(2,2) 

/AIR  ,C 

9 

- 

OATE( 2) 

/DT  , DY 

9 

- 

!*A*(2/100/2) 

/  N H  A  (  3  )  /  NO'l  (  A) 

9 

- 

REA  /PN( 3) 

/PLS(  2/3) 

9 

- 

S  V  A  R  (  1  C  > 

,  TC  /VAE(oI) 

9 

• 

HJC(2) 

~  LITERAL  FORMATS  UF  DATA  NAMES 
I.OGICAL  L«(10)  ,LD  ,LA(;3) 
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FM  rAGE 

DATA 

BLANK 

/DASH 

/NO 

,X 

z  YPS 

- 

/1H 

zlH/ 

z  2HNQ 

/ 1  HX 

z  3"YES  / 

DATA 

AIR 

,RFA 

- 

/7HAIKLIFT 

/7HN0N- 

AIR 

/ 

DATA 


— 

/ 'HRT ' 

/'CON' 

/ 'DL A '/ 

'ul>r' 

/ ' CPL ' 

— 

'DSS  ' 

/  'FLA' 

/ "FLWR'/ 

'I  AC' 

/'IPC' 

- 

'N' 

/ 'OSTICON' 

/ 'OSTIOS ' 

- 

'OSTNOCN' 

/'OSTNOS 

0 

/'PIUP' 

- 

'PSLRCON' 

/ 'PSLROS 

0 

* 

- 

'PS4RC0N' 

/ 'PSMROS 

0 

/ 

- 

'PW’CON 

0 

/'P\»DQS' 

z '°AC ' 

- 

'  D  PC  ' 

z'SA' 

/ 'SPTCON 

0 

/'SRICS 

• 

'SRNCON 

0 

/'SPNCS' 

z'TL' 

/  '  V  F  ' 

- 

'UP' 

z'VD' 

,'VF' 

/'YI' 

/  '  7, 1 ' 

- 

'DP' 

z 'DMR ' 

/ 'FAR' 

/  'FF' 

/'P' 

- 

M' 

/'LA' 

z'LP' 

z  'PP' 

/'DP  A' 

- 

'PMM' 

z  '  SC  • 

/'S#' 

/'UC' 

,'Ua' 

- 

'  4  ' 

/'X' 

/'YD' 

z'ZD' 

/'MTDC' 

>’ 

- 

'DA' 

/'DA”' 

/'FA' 

z 'FAM ' 

/ 'DCS' 

- 

'DUA' 

z ' TCS  ' 

z'TUA  ' 

/ 

L 

c 

—  RE COR T  FORMATS 

c 

2 

FDRMAT(8F9.2,8X) 

3 

FORMAT(1H1/10X/'CONSTANTS  USFD 

IN  THIS  RIJN'/^X, 

'DATE'/? 

- 

A8/2X/A 3//) 

4 

F0RMAT(2X/ A7/F12.4/1X/ A7 

/F12.4 

/4X/A7/F12 

.4) 

5 

FORMAT(I3/5X/7A9) 

6 

FORMAT(FR.O/4Fti.2/2X,Il ) 

3 

F3RMAT(8F9.2) 

809 

FORMAT (4F9. 2/4X/3I2) 

119 

F0RMAT(2CX/'EC0N0M. TC/SFNSITIVITY  ANALYSIS'/6X,' 

NUMBER'/ 

9 

F0RMAT(2V/I4/4X/2AR,A1,1T/3A8/ 

A2/2A8/A2) 

109 

FORM AT(/11X/ 'K  FACTORS: 

Ki'/F5 

.2/'  X  2'  / 

FS .2/  ' 

K3 '/ FS  . 

- 

'  K4'/F5. 2 ) 

10 

F0RMAT(/6X/ 'DESIGN  MTFF' 

/  F  3 . 0  / 

2X/'MT?D'/ 

F10.1/' 

UC'/ 

- 

F3.0/3X/A7) 

FORMAT(Il/9X/4AlO/F12.2/F5.2/l3) 

FORMAT ( I3/7X/3A10/F3.1) 

FOR  MAT(1  m1/7X/'INTERMEDISTF  MULTIPLE  SUPPORT  AGE  SUMMARY '/ 
8X,'rATE'/'>X/Afl/2X,AP//) 

FORMAT ( '  AGE  NOMENCLATURE'/ 1 1 K, "*UC'/ 4X/ 'A GF  CnST',3X/ 

'HRS  AVATL'/2X/'SGF  SETS  NEEDED '/ ) 

FORMAT (1H  /3A3/A2/1X/A5/F0.2/2X/FU.2/6X/IS) 
FORMATC/IX/'ITEM  NO.  DEMANDS/  mTTT  RFyui RET '/ 

'  *  TOTAL  SHAPE  OF  AGF  COST  '/13X/ 'MOHT'i'/ll  X/ 

' iiDURS'/5X  / ' HOP HS'/SX/'COST'/SX/' ALLOCATION'/) 
FODMAT(2X/I4/4X/F7.1,4X/FS.1/FR.1/F9.2/2(2X/C,10.2)  ) 
FO°MAT(/qX/'TOTALS'/10X/F10.1/F9.2/2Fl2.2) 
FJRMATdf'l/llX/'DEPOT  MULTTPLT  SUPPORT  AGE  SUMMARY '/ 4X/ 
'DATE'/2X/ AH/A2/) 

F(J9MAT(/26X/ 'ECONOMIC  ANALYSTS'/) 
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22  FORMAT!/ 26X, 'DEPOT '/ 6X, 'INTEPMEDI ATF',6X, 'DISCARD'/) 

550  FORM AT !//26X, 'INPUT  DATA  VALUES'//) 

23  FORMAT (IX/ 'BASE  STOCK  LEVEL',6X, I9,6K, 19,6X, 19) 

551  FORMAT(4X,A7,F12.4,2X,A7,F12.4,2X,A7,F12.4) 

24  FORMAT! '  AGE',19K, I9/6X/ T9  ) 

25  FORMAT( '  ACE  M A INT. *, 12X, 19/ 6 X/ 1 9) 

129  FORMAT!"  ITEM  NR", 7X, 'PART  NUMBER', 7X, 'NOMENCLATURE'/ 1 4X/ 

'NEXT  HIGHER  ASSEM'/) 

26  FORMAT!  '  TECH  I)  AT  A  '/  13  X/  1 9/ 6  X/ 1 9) 

27  FORMAT ( '  TRA INING '/ 14X/ I9/6X/ 19) 

28  FORMAT! '  PACKING  /  SR  I PP l NG", 4X,  1 9,6X, I  9, 6X, I  9) 

29  FORM AT ('  SAFETY  STOCK '/ 10X/ T9) 

30  FORMAT! '  LABOR"/ 17X/I9,6X/I9) 

31  FORMAT( '  SPECIAL  F ACIL IT  I ES '/ 4X/ I  9/ 6X/ 1  9) 

32  FORMAT<"  REPAIR  M ATERI AL ",7X, 19, 6X, 1 9 ) 

552  FORMAT (4X/A7/F12. 4, 2X/ A7/F12.4/) 

149  FORMAT!/) 

33  FORMAT!"  ITEM  ENTRY ',1 ?X, 19, 6X, 19 ) 

34  FORMAT!"  SUPPLY  ADMIN. '/24X,  19) 

35  FORMAT!"  PIPELINE  SP ARES "/ 7X/ 19 ) 

36  FORMAT!"  REPLACEMENT  SPARES', 34X, 19) 

37  FORMAT(/5X/'TQTAL',12X,I1Q,5X,I10/5X,I10) 

40  FORMAT(2X, 'LIMITS-  FROM  20%  TO  500%  OF  ORIGINAL  FACTOR', 

'  VALUE  WTTHIN  U '/I  OX, ' PR INT ED  AT  REVERSAL*/) 

41  FORMAT! I4/3A8, A3, F8.0/F8.0/3A1) 

411  FORMAT(2X,r3,4X,3A9,Al,F8.0,F9.0,2(5X,Al),6X,Al) 

42  F0RMAT(1F1,25X, 'SENSITIVITY  ANALYSIS'//) 

43  FORM  AT !/7X, "FACTOR", 6X,'  %  OPIG  '/6X," VALUE',! X ,' DEPOT' , 

6X, 'INTER', 4X,  'DISCARD') 

44  F!JPMAT(/7X/A7,14X/P9.2/1X/3I12/4X) 

45  FGRMAT(/t8X,F6.0,4X,F9.2,lX,3ll2,4X/) 

412  FORMAT(I8/2A8/A2,3A8/A3/2A8/A2) 

46  F(JRMAT(/21X, 'REPAIR  LEVEL  SUMMARY ',1 5  X, 'DA  TE ',  2X,  A  9,  2X,  A  3/ / ) 

47  PORMAT(3X, 'ITEM', 30K, 'UNIT', 14X, 'REPAIR  LEVEL', 5X) 

48  FORMAT(2X, "NUMBER', 3X, 'NOMENCLATURE', 13X, " PR  I CE ', 5X, 

*MT8D',3X, 'DEPOT  INTER  DISCARD"/) 

49  FORMAT!//) 

556  FORMAT  (7  X,  "NOMFNCLATURE',5X,'NFA',10X,'DATE',ljX/A8) 

6010  FORMAT!/'  CO  YOU  *ISH  AN  EXPLANATION  CF  THIS  PROGRAM? ', 

'  ( YES /NO )  ', $ ) 

6011  FORMAT(A4) 

6012  FORMATUX,  A4) 

6020  FORMAT(//20X,'ORLA  COST  MODEL'//) 

6030  FORMAT!'  ENTER  CONSTANT  VALUFS  (36  VALUES)  IN  ORDER  AS  LISTED') 
9350  FORM  AT(7X/10{A7,5X)) 

9991  FORMATUX, 10F12. 4) 

9353  F3RMAT!/7X,9!I3,9X>) 

6040  FORMAT!/) 

6050  FORMAT!/'  ANY  ADDITIONAL  CONST  ANTS/VARl  ARLES  IPR  SENSITIVITY  ', 
'ANALYSIS? ') 

6055  FORMAT!/'  MOW  M  ANY?  (LIMIT  10)') 

9992  FORMATUX,  12) 
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1620 

1621 

1622 

6060 

1640 

6065 

6070 

1710 

1711 

1712 
6080 

6090 

6091 
6100 

9354 

2955 

9352 

6110 

2000 

6120 

6140 

6130 

6151 

6152 

6153 
6160 
6161 
2997 
2993 
2999 
C 

C 

C 


C 

c 

c 


r 


Mct»owwm 


4 


FORMAT!/'  NAME  '/I3,'  ADDITIONAL  CQNSTANTS/VARI3LFS'/ 

'  (  USE  '/ 1 3/  '  LINES)') 

FORMAT (A8) 

FQRMAT(1X/10A8) 

FORM IT ( '  INCORRECT  NAME') 

FORMAT! 2X/ A8/ '  DROPPED  FPO«  ANALYSIS.') 

FORMAT!/'  FNTEP  THE  NUM8FR  OF  ITEMS  TC  RE  RUM  IN  THIS  ANALYSIS') 
FORMAT! 18X/'ENTER  ITEM  DATA') 

FQRMAT(/1X/'ENTFR  PART  NUMBER/  NOMENCLATURE/  NPXT  '»I0HEP  ', 

'ASSFMBLY'/IOX/'^OP  ITEM  NUMBER  '/ I3/4X/ '( US"  3  LINES)') 
FORMAT! 3A8/4A8/3A8) 

FORMAT! 1 X/ 3A8/1X/4A 8/1 X/3A 3) 

FORMAT!'  ENTER  MEAN  TI«E  BETWEEN  FAILURE/  *  FACTORS  (4  VALUES)') 
FORMAT!'  AND  SHIPPING  CODE  !0  =  AIRLIFT/  1  =  NON- AIRLIFT )  ' ) 
FORMAT!1X,P3.0/4E6.2/I4) 

FORMAT!'  ENTER  VARIABLES  !23  VALUES)  F JR  THIS  ITEM  IN  THE  ', 
'PROPER  ORDER') 

F0RMATI/7X/10!  I2/10X)) 

FORMAT!/'  ANY  CORRECTIONS?  (YKS/MO)') 

FORMATI7X/10IA7/5X)) 

FORMAT!'  00  YOU  WISH  TO  PUN  AN  AGE  SUMMARY  COMPUTATION?'/ 

'  IYES/NO) ') 

FO»MAT!/'  DO  YOU  WANT  AN  EXPLANATION  OF  AGE  SUMMARY?  (YLS/NO)') 
FORMAT!/'  FNTER  THE  NUMBFR  OF  ACE  SUMMARIES  TO  3E  RUN') 

FORMAT!/'  FNTER  TYPE  OF  AGF/COST/  AVA IL AB IL ITY/HOL'PS/  A‘P'/ 

'  THE  NUMBER  QF  ITEMS  YOU  HAVE  FO»  T"E  AGE  SUMMARY') 
FORMAT ! '  ENTER  ACE  NOMENCLATURE  (MAXIMUM  OF  26  CHAP)'/ 

'  AND  WUC  UNIT  CODF  (USE  2  LINES)') 

FJPMAT(3A8/A2/2A8) 

FORM AT (1 X/3A8/ A2/1 */?A8) 

FURMAT(1X/T1,3F10.2) 

FORMAT!/'  FNTER  THE  ITEM  NUMBER  AND  MEAN  TIMF  TU  TFST ' ) 
F0RMAT(1X/I4/F10. 2) 

CJR.MAT(F9.  C/4F6.2/T1/7F10.2) 

FORMat(5F10.2) 

FORMAT! 11F10. 2) 

—  OPEN  LOGICAL  DEVICES 

0P5N(UNIT=</0EVICF='DSK: ') 

OPEN  (  UNIT  =  4/ DE  V  ICE  = 'DSK:  '/DIALOG -'ORL  A.  IMP'/  »CCt’SS='Sf  OIK') 

OPEN! UNIT-10/ DEVI CF='DSK:  '/ ACCFSS  =  'SEQl  NQUT' /  CTSPUSF.r  ' DELFT F. '  ) 
OPEN! UNI T=l 2/DFVI CE='DSK: '/ ACCFSS= "SFOl M0UT'/Ur3PQSE= 'DELETE ' ) 

--  ADD  CODE  TO  ALLOW  RECOVERY  OF  STATISTICS  UPON  ABNOk^Al  EXIT 

ASSIGN  510  TO  IREEM 
CALL  KEEN! IREFJN) 

ItJ'ITPT  =  6 

WRITS! IOUTPT/6010) 

T\'nUT  =  4 
I  WORK  1  =  1 0 


A-5 


09LA 
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11 

t teQRK2=l 2 

C 

C 

*% 

—  REPLACE  WIT'!  PA N03M 

c 

REAO( INPUT, 6011) ANSWER 

12 

CALL  ASK (ANSWER ) 

c 

—  WRITE(IOUTPT,6012)A 

13- 

14 

TP(  ANSWER.  Ey.VESK  ALL 

c 

c 

—  INITIAL  V AR I AHLFS/T 

c 

15 

ICOUNT=0 

16 

00  51  N= 1, 100 

17 

AIA(N)=0.0 

18 

DAA(N)=0.0 

19- 

20 

51 

CONTINUE 

21 

DO  511  1=1,18 

22 

NVAR(l)=0 

23- 

24 

511 

CONTINUE 

25 

NVAR(1 )=31 

26 

NVAR(2)=32 

27 

N V AR ( 3  )  =  47 

28 

NVA»(4)=50 

29 

NV  4R(5)  =  56 

30 

NVAR(6)=62 

31 

NVAR(7)=64 

10/31/1980  11:36  ?'*  PACE 


32 


33 

34 

35 

36 


37 


38 

39 

40 

41 


c. 

c. 

C 

r 

C 

C 


c 

c 

c 

c 

c 

r 


C 

r 


--  ? 

WRITF( IOUTPT/6020) 

—  READ  CONSTANT  VALUES 

CALL  DATEV(UATF(1),DATE(2)) 

WR l TE( lOUT^T, 6030 ) 

«RTTE(  100101,9353X  1,1  =  1,8) 
W31TE( TQUTPT,9350)(VAR(I),l=l,9) 


INPUT=4 

—  REPTACE  WITH  PA  NOON  REAL 

PEA0(INPUT,*)(VAL(I),r=l,9) 

CALL  RFAL4(VAL(l),9,-2,2) 

VH  T  TE( I0UTPT,999l X VAL( l ), 1=1, 9) 
WRTTE( TUUTrT,43b3)( I, 1=10, 13) 
WRrTFC  TGL'TPT/9  360  )(V  AH  (  0,1=10/18) 

—  REPLACE  rfIT'»  RANDOM  REAL 
READ  (INPUT/*  XV  AL<  T),I  =  10,  13) 


-1— L, 
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i 

I 
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6 


42 

43 

44 

45 


46 

47 

48 

49 


50 

51 

52 

53 


54 

55-  56 

57 

58 


59 

60-  61 
62 


63 

64-  65 

66 


67 

63 

69 

70 


C 

C 

C 

C 


C 

c 

c 

c 


c 

c 

c 


c 

c 

c 

c 

c 

1600 

c 

c 

c 

c 

c 


c 

c 

c 

c 

c 


CALL  REAL4(VAL(10),9,-2,2) 

WRITE! IOUTPT,9  99l)(VAL(D,l=10,  18) 

WRITE! I0UTPT,9 353 )(  1,1  =  19/27) 

WRITE! IQUTpT,9150)(VAR(l),t=19, 27) 

—  REPLACE  WIT9  RANDOM  REAL 

REA0(INPUT/*)(7AL(I)/I=19/27) 

CALL  REAL4(VAL(19),9,-2,2) 

WRITE!  IOUTPT,9991)(VAL(D,I  =  19,27) 

WRITE! lOUTPT, 9 35 3)! 1/1=29/16) 

WRITE! IOOTPT/9 150)! VAR! I)/ 1=28/36) 

—  REPLACE  WITH  RANDOM  RE AL 

R€AO(IKPUT,*)(VAL(  0,1=28,36) 

CALL  PFAI 4!VAL!29)/9/-2/2) 

WRITE!  TUUTPT,  9991)!  VAL!  0,1  =  78,36) 

WRITE! IGDTPT/2955) 

I NPUT=4 

--  REPLACE  WITM  RANDOM  VES/NO 

PEAD! INPUT/ 6011) ANSWER 
CALL  ASK ! ANSWER ) 

IF(ANSWEP.EU. YES) CALL  CORECT!  INPUT, UWTPT,  VAL, MiSmF.H,  V  If  ) 
WHITE! IOUTPT/6050) 

I NPUT=  4 

—  REPLACE  WITH  RANDOM  YES/NP 

PEAU!  INPUT/6011  msWFR 
CALL  ASK (ANSWER) 

WRITE! IOUTPT/601?) ANSWER 
IF! ANSWER. EO.NO)CUTU  50 
WRTTE! IUUTPT/6055) 

—  REPLACE  WIT'?  RANDOM  INTEGER 

RE AD ( INPUT , *  )  l T 
CALL  INT4(IT/1/1/10) 

'WRITE!  IUUTPT/999?)IT 
T  F( IT. CT. 1 0) GOTO  1600 
WRITE! IOUTPT/16  2D  )  IT,  IT 

—  REPLACE  WTTR  RANDOM  INDEX  OF  VAR 

READ! INPUT, 162 l )(SVAR(J),J=1, IT) 

CALL  CHPVAPIVAP/SVAR, IT) 

WHITE (  rOUTPr,1622)(SVAR(  )),,J=l,lT) 

I  1  =  0 

DO  1630  1= I, IT 

DO  1635  J=l/64 
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1635 


1638 

1630 

C 

c 

c 

50 

C 

C 

C 

c 


c 

c 

c 

c 


c 


c 

r 

c 

c 


TF(SVAR<  I).SQ.VAR< )}}Gj7)  16? fi 

CONTINUE 

WRITE! IOUTPT/6060) 

IJsIJ+1 

WRITE!  taUTPT,1640)SVAR(I) 

GOTO  1630 
NV  AR !7  *I- 1 J  > -J 

CONTINUE 

—  COMPUTE  QCTCM  FOR  EACH  PASS 

WRITE! iaUTDT,6040) 

CONTINUE 

WRITE! IOUTPT,6065) 

—  REPLACE  WITH  RANDOM  INTEGER 

RE AO! INPUT/ * ) I T 
CALL  INT4!IT,1,1,19) 

WRITE! IOUTPT/9992) I T 
DO  5560  y~ 1/ IT 

WRITE! IOUTPT/6070) 

WRITE! IOUT°T,149) 

WRITE! TUUTPT,1710)K 

—  REPLACE  WITH  RANDOM  CHARACTER 

READ! INPUT, 1711 )PN,NOM,NHA 
CALL  CMAR3(PM,3) 

C  ALL  CHARM ! NON ,  4  ) 

CAf.L  C«IAPd!N'!A,3> 

—  -RITE!  I0UTPT,1712)P?!/N0V,NHA 

I P  A  S  S  =  E 

WRITE! TWOSK1/412 ) IPASS/PN^NO  1,NMA 
WRITE! IUUTPT/6040) 

WRITE! TOUTPT/bOeO  ) 

WRITE! TOUTPT/6099) 

--  ° KPT. ACE.  MTU  RANDOM  REAL/I  NTEGE° 

FEAO!  INPUT, *)XMTRF, KKI,XK2, X*r 3, XKl, LIFT 
CALL  REAL4!XMTPE,l,-2,2) 

CALL  REA»  4(XKl,t,-2,2) 

CALL  PKAI. 4!XE2,  1,-2,  2) 

CALL  REAL1!XK3,l,-?,2) 

CALL  REAL4! XK4, l, -2, 2) 

CALL  I  NT  4  !  L I ET  ,  1 , 0 , 1  ) 

WRITE! lUMTOT, 6091 )XMTBE,XF 1,XK?,XK 1,YK 4, LIFT 
WRITE!  rUtJTPT,6910) 

VRITF,!  IOUTPT/C100) 

WRITE! TU«TrT,9364)! 1,1=1,10) 

WRITE! TUUTPT, 93L?)(VAH(I),I  =  37, 46) 

INP JT=  1 


I 
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PACE 


Q 


110 

111 

112 

113 


114 

115 

116 
117 


118 

119 

120 


121 

122-  123 


C 

C 

c 

c 


c 

c 

c 

c 


c 

c 

c 

c 


c 

c 

c 

c 

c 


124 

125 

126 

127 

128 

129-  130  5550 

C 

c 

c 

131 

132 

133 

C 

c 

c 

c 

134 

£ 

135-  136 

r 

c 


—  REPLACE  WITH  RANDOM  REAL 

REA0(INPUT,*)(VAL(I),I=37,46) 

CALL  RSAL4(VAl.(37),lO,-2,2) 

WRITE(  T0UTPT,9991)(VAL(I),I=37,46) 

K  R I T  E  ( IOUT°T, 9354) (1,1-11,19) 

WR I  TF(  IOUTPT,9352)(VAR(I),I=47, 55) 

--  REPLACE  WITH  RANDOM  REAL 

PEAU(INPUT,*)(VAL(I ), 1=47, 55) 

CALL  REAL4(VAL(47),9,-2,2) 
WRITE(TOUT°T,9991)(VAL(I),I=47,55) 

MRTTEdUUTPT,  9 354  ) <  1 ,  1  =  20,  23) 

WRTTE( TOUT°T,935?)(VAR( I), 1=61,64) 

—  REPLACE  WITH  RANDOM  REAL 

RE»0(INPUT,*KVAL(I),I=61,64) 

CALL  REAL4(VAL(61),4,-2,2) 
WRTTE(lOUTPT,9991)(VAL(I),I=61,64) 

WRITE(ldUTPT,2955) 

—  REPLACE  WITH  RANDOM  YES/NO 

RE AO( INPUT, 601 1 ) ANSWER 
CALL  ASK( ANSWER ) 

—  4RITE(ICUTPT, 6 012) ANSWER 

I F( ANSWER. EU.YCS)C ALL  CORFCT( IN»UT, IOUTPT, VA L, A iJSWER 

VAR) 

WRITE! IWOhKl,2997)XMT9F,XKl,XK2,XK3,XK4,LIFT, 

(VAT  (I),£=37,43) 

WRTTE(  TW0RK1,2998)('/AL(I),I  =  44,  44) 

VIRITE(  TW0RK1,2  999)(  V  AL(  I ),  1=  49, 55)  ,(VAL(J),J=6  1  ,64  ) 
VAL(56)  =  X,VTBF/(XK1*XE2*XK3*XK4) 

OCTGM( TPASS)=V AL(31 )*VAL(3?)*VAL(46)/VAL(56) 

CONTINUE 

—  AGE  SUMMARY  COMPUTATION 

WRITE( IUUTPT, 6  04  0) 

WRITE(I0UTPT,6110) 

I NRJT=  4 

—  REPLACE  WITH  RANDOM  YF.S/NO 

PEAD(lNPUT,60ll)ANS<ED 
CALL  ASX( ANSWER) 

—  WRTTE(IOUTPT,6G12)ANSWER 
IF ( ANSWER. FQ. NO )GOTO  o5 

—  AGE  COMRUTATIN  ROUTINE 
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PAGE 


137 


138 

139-  140 
141 


142 

143 

144 

145 


146 

147 

148 

149 


150 

151 

152 

153 

154 


C 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


c 

c 

c 

c 


c 

c 

c 

c 


155 

156 

157 

158 


159 

160 
161 
162 

163 

164 


—  ITAG£=1  I NTERMEDI ATE  /  =2  DIRECT 
WRITE(10UTPT/2000) 

—  REPLACE  WITH  RANDOM  YES/NO 

READ!  I NPTJT/6011)  ANSWER 
CALL  ASK! A  MS WE °) 

—  WRITE!  IOUTPT/6012)  ANSWER 

IE( ANSWER. EQ.YES) CALL  AGFTLK! lOUTPT) 

WRITE!  IOUTPT/6120) 

—  REPLACE  WITH  RANDOM  INTEGER 

PEAD!INPUT,*)IT 
CALL  TNT4(IT/1/1/10) 

—  WRITE!IOUTPT/9992)lT 
DU  553  L=1/1T 

WRITE! TUUTpT/6040) 

WRITE!  IOnTpT/6130) 

—  REPLACE  WITH  RANDOM  CHARACTER 

READ! INPUT/6151)<(  AGE!  l/J),J=l,  2)/ 1  =  1,2  )/*t'C 
CALL  CHAR 8! AGE, 4) 

CALL  C-UR8(*UC,2) 

WRITFC  IOUTPT/6152)!! ACE!I,J),J=l,?),I=l,2),*lC 
WRITE!ICUTPT,6140) 

—  REPLACE  WITH  RANDOM  INTEGER/REAL 

READ( INPUT/*  )  IT AGE, KC, Ah,RNAU 
CALL  INT4! ITACE,1,0,9) 

CALL  RFAL4! AC,t,-2,2> 

CALL  KFAL4! AH,l,-2,2) 

CALL  RFAL4(KNAGE,1,-2,2) 

WRITE! rOUTPT,6l53)ITACE,AC,AH,RNAGE 

—  WRITE!  IOUTPT/6150) 

NAGE=RNACE 
THRS=0 . 0 
DO  57  1=1/ N AGE 

WRITE! TGUTPT/6160) 

--  REPLACE  WITH  RANDOM  INTEGtR/RFAL 

READ! INPUT/*)IPAS!I)/TTT‘H  I) 

CALL  INT4!IPAS!I)/l/l/10) 

CALL  RFAL4ITTTM!  D/1/-2/2) 

WRITE! IOUTPT, 6161) l PAS!  I),TTTM!  I) 
IF=IpAS!I) 

HR3!I)  =  (JCTGM!IP)*TTTM(l) 

TH',S  =  THPS»'(PS!I) 
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165-  166 

57 

167 

168 

169 

170 

54 

171 

172 

173 

55 

174 

175 

56 

176 

177 

178 

179 

180 

131 

182 

183 

184 

185 

186 

187 

188 

60 

189 

190 

61 

191 

62 

CONTINUE 

XNAS  =  ATNTITHRS/AH«-. 99999) 

NAS=XN  AS 

IF (ITAGE-2 >54^55/65 
WRlTE(I0UTDT/49) 

WRITE! TOUTPT/13)DATE(l )/DATE(2) 

GOTO  56 

WRITE! IOUT°T/49) 

WRITE! IOUT°T/ 19 )OATE!l )/0ATE!2) 

WRITE! IOUTPT/14) 

WRITE! IOUTPT/15)!! AGE! I/J)/J=l/ 2)/  Dl/2)/ 

WUC!  D/AC/AH/NA3 

WRTTE! rOUTPT/ 16) 

TSHAR=0.0 
TFPAC=0.0 
TXT  A=0 .0 
PO  63  I  - 1  /  <*/  A  G  E 

IP=IPAS!l> 

E»AC=HRS!I )/THPS 
SHARF=FRAC*AC 
XI A=SH ARE*XNAS 
FR  AC=FPAC*100.f> 

IF! ITAGE>2)60/61/65 
AIUTP)  =  XIA 
GOTO  62 
OAA!IP)=XIA 

»'ElTE!IUUTpT/17)IPAS!I),yCTGw(I?)/TTTV!l)/ 
HRS!  I)/FRAC/S(iARF,XD 


192 

193 

194 

TFVAC=TFRAC*FRAC 

TSFAR  =  TS  HA  R-*- SHARE 

TX1 A=TXI AfrTl A 

195- 

197 

196 

63 

CONTINUE 

WRITE!  lOUTPT/13)TiiRS/TFRAC/TSHAR/TXI  A 

198- 

199 

553 

C 

C 

C 

65 

C 

C 

CONTINUE 

—  URL  A  PASS  ROUTINE 

200 

REWIND  IWORK1 

—  WRITE  CONSTANTS  FOR  RUN 

201 

202 

203 

C 

69 

WRITE! TUUTPT/49) 

WRITE! IOUTPT/3)DATF!l)/UATE!?) 

WRTTF!  TUUTPT/4)!V?'>!T)/VAL!I)/T  =  l/  36) 

204 

70 

READ! IWOPK1/412/END=500)IPASS/PN/NOM/NMA 

205 

206 

207 

208 
209 

71 

PE AD! 1 WOPK 1/ 2997) XVT9F/ XKI/XK2/XK 3/XK 4/LIFT/ !VAL( 
RCAD!IWURK1/2998)!VAL! I)/ 1=44/48) 

READ!  IWOFK1/ 2999)!  VALID/ 1  =  49/55  ),!VAL(J),J  =  fc  1/6  4 
VAL!56)  =  VMTPF/!XKl*XK2*XX3*XK-4) 

IF! LI FT >73/72/73 

210 
211 
•  212 

72 

OSTC  =  V  AL{ I 7) 

OSTO=V  AL ( 13) 

3R=VAL! 26) 
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213 

SVl=VAL(27) 

214 

TC=AIft 

215 

GOTO  74 

216 

73 

0STC=V  AL( 14) 

217 

GSTO=VAf.(  15) 

218 

SR=VAL!2B) 

217 

SRl=VAL! 79) 

220 

TC=RE  A 

221 

74 

WRITE! IOUTPT/49) 

222 

WRITE! TQUTPT/119) IP  ASS 

223 

WRITE! IOnTPT/129) 

224 

WRIT£!lGUTPT,9)IPASS,PN,NGW,NhA 

225 

WRITE!  TQUTPT,10)XMTBF,VAf.!56),VAt.!5C),TC 

226 

WRITE!  TOUTPT,109)XKl,XK2,XK3,XK>i 

227 

ASSIGN  75  TG  JUMP 

228 

c 

C 

GGT'J  100 

—  PRINT  ROUTINE  FOR  ECONOMIC  ANALYSIS 

229 

V* 

75 

WRITE! IOUT°T/ 550) 

230 

WRITE! TGUTaT,551)!VA»< I >, V AL ! I >, T=37, 54 ) 

231 

WRITE! iaUTPT/551)VAR!55),VAL!55)r! VA9!I),VAL(  1)/ 1  =  61,0?) 

232 

WRITE!lUUTPT,552)!YAO! I) , VAL! I ), 1=63, 64 ) 

233 

WRITF! IUUTPT,49) 

234 

WRTTE! IQUTPT, 2 l ) 

235 

WRITE! IOCTPT, 27) 

236 

OJ  60  1=1,12 

237 

IF! 1-3 )76,76,77 

239 

76 

KU!I)  =  0V!  I) 

239 

KF!I)=EV!I> 

240 

KT! I )=TV! I ) 

241 

77 

TF!I-ll >78,78,79 

242 

78 

EO(l )  =  nv! I  ) 

243 

FF! I  )  =  FV!  I  ) 

244 

got;)  bo 

245 

79 

FIJ!l)  =  DV!  I) 

246-  247 

30 

CONTINUE 

248 

WRITS!  IUUTPT,23)«fO!8),KF!5),KT!3) 

249 

WRITS!  10UTPT,24)NJ!2),KF!1) 

250 

WRITE! IQUTPT , 25 )S J! 3),KF!2) 

251 

WRITE! IOUTPr, 26 )K0! 4),KF!3) 

252 

WRITE! IGUTPT,2R)K0!5),KF!4) 

253 

WRITE! ruUTPT,28)KO!6),^F!7),XT! 2) 

254 

WRITE!  TOT'TPT,2  ))KD!7) 

255 

WRITE!  IC'JTPT,30)FD!9),XF!6) 

256 

WRTTE! TUUTPT,31 )ED! 10),FF! )) 

257 

WRITE! TOUTPT,32)*L( 11 ),KF! 10) 

258 

WRITE! 10UTPT,33)KU!1?),KF(11) 

259 

WRITE!  lOL'TPT,  3  4 )  rF!  8  ) 

260 

WRITE!  lU'.tTPT,  J5)KC!  1) 

261 

WRITE! IOPTPT, 36)FT!1) 

262 

W°l  TS!  10UTPT,37)X[)T,KFT,KTT 

C 
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263 

264 
265- 

267 

268 

269 

270 

271 

272 

273 

274 

275 

276 

277 

278 


279 

280 
281 
282 
283 

34 

*85 

286 

287 

283 

289 

290 

291 

292 

293 

294 

295 

296 

297 
299 

299 

300 

301 

302 

303 

304 

305 

306 
)7 
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266 


C  —  WRI ?E  TO  WORK  TAPE  THE  REPAIR  LEV'LL  SUMMARY 

C 

DO  82  K-l/3 

QUT(K)=PLANK 

82  CONTINUE 

I F(KFT-KPT)37/ 97/95 
85  IF(KTT-KCT)89/89/8R 

96  IN0EX=2 

GOTO  95 

87  IF(KTT-KFT)39y99,36 

88  IND£X=l 
GOTO  95 

89  INDEX=3 

95  OUT( INDEX ) =X 

WRITE(  I40RN2  / 4 1 )  If’  ASSy  NOM  /  V  AL(50  )/VAL  (56 )#  (OUT  ( I )  /  1  =  1  /  3  ) 
f C0UNT=ICJUVT*1 
GOTO  200 
C 

C  —  COMPUTATION  ROUTINE 

C 

100  TXX=OCTCM(TPASS)*VAL(16)*12.0 

TV(1)=TXX*VAL(50) 

PJCC0N=VAL(19)«-VAL(17)*VM.  (21)*SR 
PSCUS=VAL(20)+VAL(ld)»VAL(22)*SRl 
PXX  =  VAL(2)*PSCCON«-(1.0-V\L(2))*PSCOS 
TV(2)=TXX*VAL(5l)#r>XX 

l)STO=QCTGM (IPASS)*(V4L(2)*OSTC+USTO*(  i.  0-VH L(2))) 
0STX=(0ST0*SQRT(3.0*0STL>) 

TV(3)=VAL(60 )*OSTX 

C 

FV(1)=AIA(IPASS)*VAL(64)*VAL(6))/VAL(11) 
FV(2)=VAL(39)*VAL(lo  )*  (  M  A  (  IP  ASS  HV  AL  (6  4  )  ) 

FV(  3)  =  VAT,(  4?)*VAL( )0)/VAL(ll ) 

FF(4)=(1.0*VAL(34)*(VAL (lb)-1.0))*(VALf52)*VAL(55)* 
(VAL(36)*40.9*VAL(8>)) 

A  =V  AL(  45  )  *VAL(  43 )  *  l  2.0*OCTGM(  IPA  SS  ) 

EG7=4.4*SQ°T(A) 

IF(£OU-A)lCd, 120/120 
100  A12=A/ 12.0 

fF(t'OU-A12)110,n0,13C 
110  F9Q=  Al 2 

GOTO  130 
120  EU7=A 

130  8RC  =  '1CTGM(  IPASS)*VAL(1  ) 

PV(5)  =  VAL(50)*(RRC*S'JRT(3.0*RRC))*VAL(4d)*(1.0-VAL(45))* 
OSTX^EOg 


F  V  ( 

6) 

=TX**VAL( 

47 ) *V AL ( 9) 

PY( 

7) 

=TXX*VAL( 

49 ) *  nXX 

FV( 

3) 

=VAL( 16)* 

VAL(25)*(V4L 

( 43 )*V  AL( 44 

)) 

FV  ( 

9) 

=  VA(.  (40) 

FV  ( 

10 

)=QGTGM(  I 

R  ASS) *1 2.0*V 

AL  (  1 6  )  *V  AL  ( 

49  ) 

FV{ 

11 

)  =  (  ”AI.(  44 

)-l.0)/VAL(l 

t)*(VAL(10) 

♦  (VALdf  )-1.0)*VAL(?4))* 

VAL(  43 

)/VAL(ll  )*(  V 

*L(9)*(VAL( 

16)-1.0)*VAL(23)) 
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C 

DV(1)=QCTCV( IPASS)*VAL(5)*VAL(50) 

DV(2)  =  (UAA<IPASS)«-VAL(62)*7AU61))/VAL(  11) 
DV(3)=(VAL(3R)*VALC16)*(CAA(IPASS)+VAL(6?)))/VAL(11) 
DV(4)  =  VAL(41)*VAT.(30>/?AL(  11) 

DY(5)=(1»0+VAL(33)*(VAL(16)-1.0))*(V4L(53)*V>L(54)* 

(VAL(35)*40.0*VAL(4)))/VALC11) 


313 

DV(6)=TXX*2.0*VAU51)*PXX 

314 

Otf(7)=QCTGH(IP 4SS)*V«L(6)*VAL(50) 

31S 

DV( 3)=TV (3) 

316 

0V(9)=TXX*VAL(47)*VAL(4) 

317 

Dtf(10)=VAL(37)/VAL(ll) 

318 

DV( 11)=TXX*VAL(48> 

319 

DY( 12)=F V ( 11 ) 

320 

KTT=0 

321 

KFT=0 

322 

KDT=  IF IX(0V{12)) 

323 

DU  140  1=1/3 

324 

KTT=KTT* IFIX(TVCI)) 

325- 

326 

140 

CONTINUE 

327 

00  150  1=1/11 

328 

KFT  =  XFT*-IFtX(FV(l)) 

329 

KOT=rDTM?IX(OV(I)) 

330- 

331 

150 

CONTINUE 

332 

C 

C 

GOTO  JUMP/ (75/ 215/715  ) 

333 

200 

Ntr=o 

334 

LSA=10 

335 

V AL(S7)=0W(2) 

336 

VAL(53)=DV( 3) 

337 

VAL(59  )=FV(1 ) 

338 

r 

VAI.(60)=FV{2) 

A- 

C 

C 

218 

—  PANK  THF  FCON  VALUES 

339 

KSFN(1)=F0T 

340 

ICSEN(2)=rFT 

341 

KSSNC J)=FTT 

342 

OJ  335  1=1/3 

343 

»!U«K(I)  =  I 

344 

i«=r*3 

345 

FSFM(I'f)=XSEN(I  ) 

346- 

347 

335 

CONTINUE 

343 

00  3100  10=1/ 2 

349 

k=ib*i 

350 

DU  3100  I7-K/ 3 

351 

IF(KoEN(  T9)-FSFfi(IZ)  >3100/  3 IOC,  304 

352 

304 

MULC=K5FN( 19) 

353 

XSFN(I9)=KSF.F(IZ) 

354 

F  SEN(  1  T.)=HOLD 

355 

MOLU  =  fiUPf(  IB) 

356 

NU*«K(  IP)=NUMK(  12) 

308 

309 

310 

311 

312 
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357 

NUMK( IZ)=H0LD 

358- 

360 

3100 

f 

CONTINUE 

c 

—  SENSITIVITY  ANALYSIS 

361 

2360 

nv=mvm 

362 

IF(NVAR(NV))70,70,2365 

363 

2365 

IC=NVAR(NV) 

364 

C=VAR( 1C) 

365 

ORIG=VAL(IC) 

366 

PCT=0. 90 

367 

00  2300  TP=1, 48 

368 

CX=ORIC*PCT 

369 

VAL(IC)=0X 

370 

2370 

OCTCM<  IPASS)=VAL(31)*VAL(32)*VAL(  46)/VAL(‘>6) 

371 

ASSIGN  215  TO  JUMP 

37  2 

GOTO  100 

c 

c 

—  REVERSAL  ANALYSTS 

373 

215 

JSFN(1 )=KUT 

374 

JSEN( 2) =KFT 

375 

JSFN(3)=YTT 

376 

00  210  1=1,3 

377 

NUMJ(I)=r 

379- 

379 

210 

CONTINUE 

380 

300 

DO  310  IP=l,2 

331 

K=I8*1 

382 

DO  310  IZ=K,3 

333 

305 

IF(JSFN( IB )-JSEN( 17)  >31 0,310 

384 

306 

HOLD=JSEN ( IP ) 

385 

JSEN(in)=JSFN(I ') 

386 

JSFN  ( 17.  )  =  HOLO 

387 

HOLD  =  NUyj( IP  > 

388 

NUMJ ( IB)=NUMJ ( TZ) 

389 

NUMJ(I7)  =  :IOLD 

390- 

392 

310 

CONTINUE 

393 

I F<NUMK( 1)-NUMJ( 1)) 320, 228, 320 

394 

228 

TF(IP-3)322,222,229 

395 

222 

PC?=PCT*  0 . 90 

396 

GOTO  2300 

397 

322 

PCT=PCT-0„ l 

398 

GOTO  2300 

399 

229 

PCT  =  PCT«-0.1 

400 

GOTO  2300 

401 

320 

IF(IP- 3)375, 375, 369 

402 

375 

PCT=PCT*0.09 

403 

GOTO  340 

404 

360 

PCT=PCT-0. 39 

405 

340 

OX=JRIG*PCT 

406 

V  AL( IC )=0  X 

407 

GCTGM<  rPASS)=VAL( 31 > * VSL ( 32) * VAl( 46)/VAL(56) 

'408 

ASSIGN  715  TO  UMP 

MCDOMft I.  DOUOL4 


A  15 


410 

C 

C 

715 

411 

412 

413 

414 

415- 

416 

712 

417 

700 

418 

419 

420 

705 

421 

706 

422 

423 

424 

425 

426 
427- 

429 

710 

430 

431 

350 

432 

355 

433 

434 

325 

435 

436- 

437 

2300 

438 

439 

440 

C 

400 

441 

4009 

442 

443 

444 

445 

4010 

446 

425 

447 

448 

449 

450 

451 

500 

452 

C 

C 

c 

505 

453 

454 

455 

456 

457 

—  CHECK  FOR  REVERSAL 

JSFN(1 )  =  KOT 
J3EN! 2)=Kf T 
JSPN(3)=KTT 
TU  712  1=1/3 

NUMJ(I)=I 

CONTINUE 
DO  71 C  10=1/2 
K=  T  ti+1 

no  710  r7=K/3 

IF!JSEN(  ID)— ,JSE*J(I7))710/7l0/7Cti 
HOLD=JSEN ( IB ) 

JSSN(IB)=JSEM(U) 

J5PN(IZ)  =  Hni.O 
TJL0=NUVJ(I8> 

NUMJ(lrf)=f!UM.J(IZ) 

NUMJ ( IZ)=HOLD 

CONTINUE 

[F(NUMK(1)~NHHJ(1)>40 0/350/400 

TF{TP- 8)3 55/355/325 

PCT=PCT-0.01 

GOTO  340 

PCT=PCT«-0.01 

COTO  340 

CONTINUE 
VAL(IC)=OKrC 
COTO  2360 

l F(LSA-IO) 4009/4007/ 4010 
WRITE!  TOIJTPT/49) 

•< R I  TF(  IUUTnT/42) 

WRTTFf  TOUTPT/40) 

WR rTF( IOHTPT/4  3) 

PCT=PCT* 100.0 

WRITE!  lUUTPT/4  4)C/ORIG,(KSEN(K)/K=4/6) 

LSA=l.S  A*l 

WHITE! IOUTPT,45)PCT/nx,KOT/KFT/KTT 
VAT.(  lC)  =  nplG 
GOTO  2360 
REWIND  TWOWK2 

—  W*>I  TF  REPAIR  LEVEL  S  U  M  M  A  R  y 

WRITE! IUOTPT / 4  9 ) 

WRITE! TGPTPT/46 )PATE!l  )/PATE!  2) 

WRITE! IUUTPT/47) 

WRITE!  IUUT°T/4‘1) 

DO  :>0d  I=l/ICO'INT 

PE AO! IWORK2/41/END=510)IPASS/NT1/VAL!50),VAL!56) 

!OUT!K)/K=l/3) 


A-16 


ORLA 


MDftC  SEGMENT  Xf.ATGQ 


10/31/1910  11 s 36  FM 


°A 

458  «RTTE( TOUTPT/4 1 1 ) IP ASS,N0M, V AL( j0 ) , V AL(5 6) , 

(OUT(L)#L=l/3) 

459-  460  508  CONTINUE 

461  510  STOP 

462  ENO 


A-17 


MD4C  Segment  Xlator 


10/31/1980  12:51  N'«  Page 


4 


ORLA 


Sejment  Reference 


1.  CO-13) 

2.  Cl  3-20) 

3.  £20,16-20 ) 

A.  £20-24) 

5.  £24,21-24) 

6.  C  24-55) 

7.  CS5-60) 

R.  £60-61,83-122) 

9.  C122-130) 

10.  C  1 30,  d.6-1 22) 

11.  C 1 30-1 35) 

12.  £135-136, 200-204) 

13.  C  204-209) 

14.  C 209,216-228,279-294) 

15.  C294-296) 

16.  C 2 96-29 8, 3 00- 3 26  ) 

17.  £326,323-326) 

18.  £326-331) 

19.  £331,327-331) 

20.  C  331-332) 

21.  £332,229-237) 

22.  £237-241) 

23.  £241-244,246-247) 

24.  £247,236-237) 

25.  £247-266) 

26.  £266,263-266) 

27.  £266-267) 

28.  £267,271) 

29.  C271, 274-278, 333-347) 

30.  £347,342-347) 

31.  £347-351) 

32.  £351,358-359) 

33.  £359,350-351) 

34.  £359-360) 

35.  £360,348-351) 

36.  £360-362) 

37.  C362,204) 

38.  £362-372,279-294) 

39.  £351-359) 

40.  £271,269-270,275-278,333-347) 

41.  £267-263) 

42.  £263,274-278,333-347) 

43.  £263,272-273,275-278, 333-347) 

44.  C241, 245-247) 

45.  £237,241) 

46.  £332,373-379) 

47.  £379,376-379) 

48.  £379-383) 

49.  C  333,390-331 ) 

50.  £391,  332-393) 


wcoowwm 


qL 


A  18 


HD AC  Segment  Xlator 


10/31/1980  12:51 


nage 


RLA  Segment  Reference 

51.  €391-392) 

52.  C392, 380-383) 

53.  C  392-393) 

54.  C393/401) 

55.  r401-40 3, 405-409/279-294) 

56.  €401,404-409,279-294) 

57.  C393-394 ) 

5R.  €394,397-398,436-437) 

59.  f 437/ 367-372/279-294) 

60.  €437-439, 361-362) 

61.  C  394-396, 436-4  37 ) 

62.  1394/399-400/436-437) 

63.  C 383-391) 

64.  €332/410-416) 

65.  €416/413-416) 

66.  €416-420) 

67.  €420/427-423) 

68.  €428/419-420) 

69.  €428-429) 

70.  €429/417-420) 

71.  €429-430) 

72.  €430/440) 

73.  C440-450/ 36 1-362) 

74.  C440/445-450/361-362) 

75.  €430-431) 

76.  €431-433/405-409/279-294) 

77.  €431/434-435/405-409/279-294) 

78.  €420-428) 

79.  €296/300-336) 

80.  €294/299-326) 

81.  €209-215/221-278/279-294) 

82.  €204/451-457) 

83.  €457-460) 

84.  €460/456-457) 

85.  €460-4611 

*  86.  €457/4611 

87.  €135/137-139) 

88.  €139-166) 

89.  €166/157-166) 

90.  €166-169) 

91.  €169-172,175-187) 

92.  €187-189,191-196) 

93.  €196,191-197) 

94.  €196-199) 

95.  €199,143-166) 

*  96.  r 199-204) 

97.  €197,190-196) 

*  98.  €187,200-204) 

99.  €169,173-187) 

100.  €169,200-204) 


A  19 


MD  AC  segment  Xlator 


10/31/1990  1 2 : '5 1  NM 


P  i'jc 


3PLA  Segment  Reference 


101* 

T 139/141-166 ) 

102. 

C122, 124-130 ) 

103* 

C  60/62-64) 

104. 

C64-65, 62-64) 

105. 

164,66-71) 

106. 

C7 1-72,79-81 ) 

107. 

£81,69-71) 

108. 

C  61-122) 

109. 

C7 1^73-7  4 ) 

110. 

£74,70-71) 

111. 

£74-78,80-81) 

112. 

C55/57-60) 

113. 

C l 3, 15-20 ) 

-  MO  C$$MONTTflR  for  MODULE  *ORL A  '  - 

AGETLK  Segment  Reference 

I.  CO-21 

-  MO  C$$MONITOR  for  MODULE  'AC.ETLK  '  - 


XPLAIN  Segment  Reference 

1.  CO-7 J 

-  MO  C$$MON I TOR  for  MODULE  'XPLAIN  '  - 


CQRECT 


Segment  Reference 


1.  CO-7) 

2.  C7-8,13-16) 

3.  C 1 6-17/0-7) 

4.  116,18-191 

5.  C7,9-10) 

4.  110,6-7) 

7.  U0-12,2-7) 


-  MO  C$$MON!TOR  for  MODULE  'CPRFCT  '  - 


oa  rev 


Segment  Reference 


1.  CO-31 


-  MO  C4SMONIT02  for  MODULE  'UATEV 

/ 


/MdMMrJMTli  001/014* 


A  20 


Appendix  B 
APTS  OUTPUT 


*  » 


MUM 


1. 

2. 

3. 


truns 


o  . 

7. 

a 


•Praijr  a>n 


CiS«> 


9. 

1C. 

11. 

12. 

1 

14. 

15. 
It. 
17. 
1«. 
1°. 
20. 
21. 
22. 


66.  67  nc 


k>6.  >t 


MCMMWU  OOUOIM 


i 

l 

.) 

l 

3 


84.0?  Pc 


Case 


13.23  Pc 


0 


S6.96  Pc 


1 

1 

0 

2 

1 

3 

2 

1 

1 


1 

0 

2 

1 

2 

0 

1 

? 

3 

2 

3 

2 

3 


30.77  Pc 


B  2 


Case  3 
33.33  nc 


0 

3 

1 


86.9fc  Pc 


1 

1 

0 

2 

1 

3 

2 

1 

1 


80.77  Pc 


Sua>a  ary 
100.00  Pc 


0 

1 

2 

3 

2 

3 

2 

3 


36.96  PC 


0 

b 

3 

6 

0 

3 

6 

q 


t/ 

9 

t. 

1 


88.  16  Pc 


Jjc.  O'  wW?OUU 


4&IN 


Segment 


1.  r  0-11 ) 

2.  rit/5-11) 

3.  C 11-1 23 

-  MO  C$$MONITOR  for  ^H-JULE  'MAIN 


TRIASG 


* 


* 


* 


1.  CO-3) 

?.  r3-i/34) 

3 .  C34-77) 

4.  C  37/2- 3 ) 

5.  C37-3d] 

6.  C  34/ 36-37) 

7.  rj/5-8) 

8.  C3-11) 

9.  1 11/7-8 ) 

10.  t l 1-1 3) 

11.  C13-IG) 

12.  C13-D/34) 

13.  C 1 3/ 20-23 ) 

14.  C  27/ 20-22) 

15.  C22-',7) 

16.  C  27-20/32-33) 

17.  C33/23-77) 

IP.  C33-31) 

19.  C27/2)-31) 

20.  t31/2)-31) 

21.  C3 1-33  ) 

22.  C 1 3/ 1 5-19) 

73.  C8/10-U) 


-  MO  C$$RON  ITHR  for  MOTJLE  *TRI  AN3 


Segment 


Reference 


Reference 


B-3 


Wuaber  of  trials(T)= 
Value  of  XI: 


3 


Trial  Statistics 


xt  n  :  l 

XC  ?1  :  1 

Vuaber  of  Xl(H)=  2 


B-4 


IJj*!  t 

"  j  j« 

C 

3  as* 

O 

Hu mi ary 

66.07  Pc 

33.  33 

Pc 

33.  33 

Pc 

100.03  Pc 

1. 

i 

3 

0 

1 

C  m 

1 

1 

■3 

i 

4. 

3. 

J 

0 

1 

\ 

A. 

TRlA'r, 

•37.61  Pc 

86.96 

Pc 

36.  >6 

Pc 

36. 3ft  Pc 

1  . 

l 

l 

1 

3 

• 

1 

1 

1 

3 

-> 

• 

) 

a 

3 

n 

*. 

2 

2 

6 

C 

•  • 

1 

1 

1 

3 

r. 

J 

3 

3 

3 

7. 

2 

2 

6 

« 

2 

1 

6 

9. 

1 

! 

l 

3 

10. 

<_ 

> 

4. 

2 

6 

11. 

0 

1 

1 

2 

i r. 

3 

0 

r> 

13. 

~ 

2 

2 

ft 

J4. 

A 

1 

1 

3 

lb. 

> 

2 

/ 

6 

1*. 

) 

9 

0 

3 

17. 

1 

l 

1 

3 

«  o 

A.  •  • 

2 

*> 

<- 

6 

19. 

3 

3 

3 

V 

~>r. 

k.  -  • 

*■*, 

2 

<: 

6 

21. 

3 

3 

> 

J 

*j 

^ 

_ 

7 

2 

6 

7°. 

1 

3 

3 

9 

•Progr  vi 

P.0.7  7  -'c 

P  ^  .  7  7 

Pc 

37.77 

Pc 

38.45  Pc 

B5 


■J  O  «.*>  Ui 


HfcIM  Segment  Reference 

1.  CO-1!) 

?.  ni/S-in 
3.  rii-121 

-  m  cj^mijnitdp  for  mlijl6:  'mun  '  - 


TRIAfIG  Segment  Reference 

1.  CO-3) 

2.  r  3-4, 34 ) 

*  3.  C34-37 ) 

4.  C  37/ 2- 3) 

5.  C  37-38  1 

f.  C31/3&-37) 

7.  C3/5-3) 

8.  C  3-11) 

9.  C 1 1 , 7-8  ) 

10.  C 1 1-1 37 

11.  C13-18) 

*  12.  C 13-1 4, 34) 

13.  Cl  8/ 20-22) 

14.  C  22/20-22) 

15.  C 2 ')-?7  ) 

*  If  .  C  27-2  3/  32-3  3) 

17.  C33/23-77) 

13.  C  3  3-34 ) 

19.  r  27/79-31) 

20.  C  31/ 77-31) 

21.  CM -33) 

22.  Cl  3/ 13-13) 

23.  C 3/ 10-11) 

-  N3  C4 ?,‘*0N ITO?  for  'n.lULS  *T9IANG  '  - 


Case 


7  Case 


3  Case 


9  Suaaary 


MAIN 

1. 

2. 

3. 

TRIANG 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
21. 
22. 
23. 


•Program 


66.57  Pc 

1 

1 

0 

86.96  Pc 

1 

1 

0 

2 

1 

3 

2 

1 

1 

2 

1 

0 

2 

1 

2 

0 

1 

2 

3 

2 

3 

2 

3 


84.82  Pc 


33.33  Pc 

0 

1 

0 

78.26  PC 

1 

1 

0 

2 

1 

3 

2 

0 

1 

2 

0 

0 

2 

1 

2 

0 

1 

2 

3 

2 

3 

2 

3 


73.03  Pc 


33.33  Pc 

0 

0 

1 

86.96  Pc 

1 

1 

0 

2 

1 

3 

2 

3 

1 

2 

1 
0 

2 
1 
2 
0 
1 
2 
3 
2 
3 
2 
3 


80.77  Pc 


100.00  PC 

1 

2 

1 

86.96  Pc 

3 

3 

0 

6 

3 
9 
6 

4 
3 
6 
2 
0 
6 
3 
6 
0 
3 
6 
9 
6 
9 
6 
9 


83.46  Pc 


\ 


B  8 


UIN 


5e.j*ent  Reference 


1.  CO- 1 1  ) 

7.  rll,'i-ll) 

3.  r 1 ! - 1 2  3 

*n  CS* 'lUMTO#  for  '»fJI)ULr: 


TRUVO 


* 


NQ 


l.  ro-3) 

?.  n-i,3?> 

3.  r34-37) 

4.  f  37,3-3) 

5.  137-311 

6  .  C  3  1/ 33-37) 

7.  C3,3-3) 

8.  3 - 1 1 ) 

a.  C 1 1 / 7- t ) 

10.  C 1 1-13) 

11.  Cl  3-1;) 

1?.  n<?-19,-M) 

13.  ri3/>u-22) 

M.  C??,^-??) 

15.  C  2?-?7) 

16.  C?7-os,  J3-3-J) 

17.  133,73-27) 

18.  C 33-^  * ) 

l?.  C27,2.<-31) 

70.  C31/2'»-Jl) 

21.  C  3  1- 3 J ) 

77.  n  .3,13-19) 

23.  15,10-11) 

C00"nvlTnS'  for  M'lU’H.K 


*T  ^  1 1.10 


Sequent  Reference 


WCPOWWU  0OVO14S 


L(©^ 


S  9 


Trial  Statistics 

(lumber  of  trials(*)=  9 

Value  of  Xi: 

XC  11  :  l 

xc  ?i  :  i 

XC  31  :  l 

xc  43  :  4 


Number  of  Xi(N)=  1 


Appendix  C 

OUTPUT  FROM  A  CONSTRUCTED  CASE 
AND 

CONSTRUCTED  CASES  LISTINGS 


OUTPUT  FROM  A  CONSTRUCTED  CASE 


*UIN 

t. 

2. 

3. 

TRUNS 

1. 

2. 

3. 

4. 
5  . 
6. 

7. 

8. 
o. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
31. 
22. 
23. 


*Pr ogr am 


Case  1 

66.67  1'c 

1 

0 


73.91  Pc 

1 

1 

1 

/ 

1 

3 

2 

-» 

1 

2 

0 

1 

1 

0 

1 

1 
0 
1 
) 
9 
0 

2 
1 


73. 9«  Pc 


Summary 
66.67  ?c 

1 

0 

1 

73. PI  Pc 

1 

1 

1 

2 

1 

3 

2 

2 

1 

2 

0 

1 

1 

0 

1 

1 

0 

1 

0 

0 

0 

2 

3 


7  3.03  Pc 


C-2 


MAII 


Se7ment  'aference 


ro-io) 

r in, 5-10) 

no-it  j 


N3  C$$"TMITT?  for 


TRf  AN  I 


0«^7Tuint  r,|!f(*r«[ice 


1 . 

r<>-3) 

2. 

C3-'1,  3 1.) 

.3. 

f  iS-'O) 

4. 

n^/?-3) 

0. 

C  3  7-  i  o  1 

b. 

CIS/ 3b- >0) 

7. 

r  3/5-0 

3. 

L  3-1  1 ) 

0. 

r 11,7- 3) 

1C. 

r li-i 3 ) 

11. 

C  13-1 n 

12. 

C17-27, 3b) 

13. 

Cl  ),  71-23) 

1<1. 

r27/ni-'’  3) 

lb. 

C  2  3-20 

1  *. 

r?'>-30,3'l- 

n. 

c33/?4-:; )) 

ip. 

ric-’b ) 

10. 

CJ7,  Jl-3  3) 

3C  . 

C3  3,31  -  13) 

1  1 

S.  x  • 

r  J3-30 

*  «  • 

m,  ib-i7) 

23. 

C  1/10-11) 

Monitor  I're.li'rjf  2  - 
Proc ir i to 
1.  f 

?.  '  m 

3.  T'-'V 


Tyo«*  ‘linin'in 

Int?  j»r  1 

::.e  j1  . 000000000000000:>30 

r«il  .000000000700000'’ *00 


“ 3\i nu  a 

i 

. &  30000001 OOOOOOR*^ 
.003000000000000E4-0 


Trial  Statistics 


Muafrer  of  trials(7)=  1 
Value  of  Xi: 

Nunoer  of  Xt («)=  0 


*fCOO«W*U  OOVOLAt 


C-4 


CONSTRUCTED  CASES  LISTINGS 


C 

C 

c 


10 


c 

101 

102 

r 


program  main 

IMPLICIT  IlfTEGBR!  A-Z) 

INTEGER  IPO) 

REAL  »(4/3)/R 

DATA  I/fliO/4.0#3*0/5«0#2«0/4«3/0«0/7»l#9tO/0«0/11.0#l2«C 

0P£M( UNIT=20  #0BTIC5=  mUSKS *#FTLE='BRAD« RES'#  ACCESS* 'SEQOUT') 

*1=3 
HH=A 
M  =  1 

DO  10  K=1#M 

WRITS! 20# 101)K#!(A!I#J)#J=l#!O#  1=1, KV) 

TALL  TRIANG!TP,A#N) 

WRITE! 20# 102 )F# ! !A! I# J)# J=1#N )# I=1#N*0 

CONTINUE 

STOP 

FORMAT!'  BEFORE  TRI ANGULAR I Z AT ION  ! '#  II# '  ) '//3(3JC#P20.  10)) 
FORMAT!  '  ATTER  TRIAflfCOLARIZATIOM  ('#11#  ')  '//3(3X#F20.10)> 

END 


SMUQUnVE  TRIANG(IP/A#N) 
IMPLICIT  I Hf BGER( A-Z) 


C 

utecEa  ip(3) 

«;\L  A( 4/ 7)#T 

OEM.  «3VT  /TWQ4 

C 

0- 

1 

i r<  n>=i 

2 

DO  6  K=1,N 

C ? TO ?- 1 7T E GE P ( A) ; 

3- 

4 

TF(K.gQ.N)  C'3TQ  5 

5 

f:pl*K«-l 

6 

U  =  K 

7 

DO  l  I=KP1,N 

8- 

9 

TF<AHS(A<I,K)).CT 

10- 

11 

1 

CONTINUE 

12 

IP(X)=* 

13- 

14 

mi.NB.K)  IP(N)=-IPCN) 

15 

T  =  A(M,!C) 

16 

I(m,K)sA(K/K) 

17 

MK#K)sT 

IS 

MQVT=A8S(T) 

C$S40NITn3=KFAL(MnNT)J 

19- 

20 

If(r.Etl.O)  GOTO  5 

21 

do  2  r=m,w 

22- 

23 

2 

A(I,K)*-A<I,K)/T 

24 

OH  4  JsKPl,N 

25 

T=AC«,J) 

26 

4<4,  D- A<*,J> 

27 

A(K,J)=T 

28 

TM0N=A8SCT) 

CS<?Mnt<lT99  =  a?:AL(TM0V); 

29- 

30 

IF(T.EO.O)  GOTO  4 

31 

00  3  I=KP1,V 

32- 

33 

3 

«(I,J)sA( 

34- 

35 

4 

OJNTINUE 

36- 

37 

5 

I f( A(K/K ) . EQ. 0 )  IP(K)=0 

10- 

39 

5 

continue 

40 

7ETJ»N 

41 

ENP 

